bpf: Support new insns from cpu v4
Subsystems (9)
v2 cover bpf: Support new insns from cpu v4
In previous discussion ([1]), it is agreed that we should introduce cpu version 4 (llvm flag -mcpu=v4) which contains some instructions which can simplify code, make code easier to understand, fix the existing problem, or simply for feature completeness. More specifically, the following new insns are proposed:
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offset
This patch set added kernel support for insns proposed in [1] except BPF_ST which already has full kernel support. Beside the above proposed insns, LLVM will generate BPF_ST insn as well under -mcpu=v4 ([2]).
The patchset implements interpreter and jit support for these new insns as well as necessary verifier support.
To test this patch set, you need to have latest llvm from 'main' branch of llvm-project repo and apply [2] on top of it.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://reviews.llvm.org/D144829
Changelogs:
RFCv1 -> v2:
. add more verifier supports for signed extend load and mov insns. . rename some insn names to be more consistent with intel practice. . add cpuv4 test runner for test progs.
. add more unit and C tests.
. add documentation.
Yonghong Song (15):
bpf: Support new sign-extension load insns
bpf: Fix sign-extension ctx member accesses
bpf: Support new sign-extension mov insns
bpf: Support new unconditional bswap instruction
bpf: Support new signed div/mod instructions.
bpf: Fix jit blinding with new sdiv/smov insns
bpf: Support new 32bit offset jmp instruction selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing selftests/bpf: Add unit tests for new sign-extension load insns selftests/bpf: Add unit tests for new sign-extension mov insns selftests/bpf: Add unit tests for new bswap insns selftests/bpf: Add unit tests for new sdiv/smod insns selftests/bpf: Add unit tests for new gotol insn selftests/bpf: Test ldsx with more complex cases docs/bpf: Add documentation for new instructions
Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 100 ++- arch/x86/net/bpf_jit_comp.c | 131 ++- include/linux/filter.h | 14 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/cgroup.c | 14 +- kernel/bpf/core.c | 174 +++- kernel/bpf/verifier.c | 315 ++++++-- tools/include/uapi/linux/bpf.h | 1 + tools/testing/selftests/bpf/.gitignore | 2 + tools/testing/selftests/bpf/Makefile | 18 +- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 88 ++ .../selftests/bpf/prog_tests/verifier.c | 10 + .../selftests/bpf/progs/test_ldsx_insn.c | 75 ++ .../selftests/bpf/progs/verifier_bswap.c | 45 ++ .../selftests/bpf/progs/verifier_gotol.c | 30 + .../selftests/bpf/progs/verifier_ldsx.c | 115 +++ .../selftests/bpf/progs/verifier_movsx.c | 177 ++++ .../selftests/bpf/progs/verifier_sdiv.c | 763 ++++++++++++++++++ 20 files changed, 1929 insertions(+), 158 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_bswap.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotol.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_movsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.c
Signature · 1 line
2.34.1
PATCH v2 1/15 bpf: Support new sign-extension load insns
Add interpreter/jit support for new sign-extension load insns which adds a new mode (BPF_MEMSX).
Also add verifier support to recognize these insns and to do proper verification with new insns. In verifier, besides to deduce proper bounds for the dst_reg, probed memory access is handled by remembering insn mode in insn->imm field so later on proper jit insns can be emitted.
--- arch/x86/net/bpf_jit_comp.c | 32 ++++++++- include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 13 ++++ kernel/bpf/verifier.c | 125 +++++++++++++++++++++++++++------ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 151 insertions(+), 21 deletions(-)
arch/x86/net/bpf_jit_comp.c +31 -1
@@ -779,6 +779,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* LDX: dst_reg = *(s8*)(src_reg + off) */ +static void emit_ldsx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; + +switch (size) { +case BPF_B : +/* Emit 'movsx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBE ); +break ; +case BPF_H : +/* Emit 'movsx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBF ); +break ; +case BPF_W : +/* Emit 'movsx rax, dword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x63 ); +break ; + } +emit_insn_suffix (&prog, src_reg, dst_reg, off); + *pprog = prog; +} +/* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) {
@@ -1370,6 +1393,9 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : +case BPF_LDX |BPF_MEMSX |BPF_B : +case BPF_LDX |BPF_MEMSX |BPF_H : +case BPF_LDX |BPF_MEMSX |BPF_W : insn_off = insn->off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {
@@ -1415,7 +1441,11 @@ st: if (is_imm8(insn->off)) start_of_ldx = prog; end_of_jmp[-1 ] = start_of_ldx - end_of_jmp; } -emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +if ((BPF_MODE (insn->code) ==BPF_PROBE_MEM && insn->imm ==BPF_MEMSX ) || +BPF_MODE (insn->code) ==BPF_MEMSX ) +emit_ldsx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +else +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {struct exception_table_entry *ex;u8 *_insn = image + proglen + (start_of_ldx - temp);
include/uapi/linux/bpf.h +1 -0
@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
kernel/bpf/core.c +13 -0
@@ -1610,6 +1610,9 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (LDX ,MEM ,H ), \INSN_3 (LDX ,MEM ,W ), \INSN_3 (LDX ,MEM ,DW ), \ +INSN_3 (LDX ,MEMSX ,B ), \ +INSN_3 (LDX ,MEMSX ,H ), \ +INSN_3 (LDX ,MEMSX ,W ), \/* Immediate based. */ \INSN_3 (LD ,IMM ,DW )
@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \ + LDX_MEMSX_##SIZEOP: \ + DST = *(SIZE *)(unsigned long) (SRC + insn->off); \ + CONT; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS +#define ATOMIC_ALU_OP (BOP ,KOP ) \ case BOP: \ if (BPF_SIZE(insn->code) == BPF_W) \
kernel/bpf/verifier.c +105 -20
239 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +1 -0
@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 01/15] bpf: Support new sign-extension load insns
kernel/bpf/core.c static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) · line 1948
Quoted 6 lines · Yonghong Song
kernel/bpf/core.c +1
-0@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \
LDSX ?
Quoted 8 lines
kernel/bpf/core.c +8
-0+ LDX_MEMSX_##SIZEOP: \ +DST = *(SIZE *)(unsigned long ) (SRC + insn->off); \ +CONT ; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS
...
Quoted 13 lines
kernel/bpf/verifier.c +4
-1@@ -17503,7 +17580,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)if (insn->code == (BPF_LDX |BPF_MEM |BPF_B ) || insn->code == (BPF_LDX |BPF_MEM |BPF_H ) || insn->code == (BPF_LDX |BPF_MEM |BPF_W ) || - insn->code == (BPF_LDX |BPF_MEM |BPF_DW )) { + insn->code == (BPF_LDX |BPF_MEM |BPF_DW ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_B ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_H ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_W )) { type =BPF_READ ; } else if (insn->code == (BPF_STX | BPF_MEM | BPF_B) || insn->code == (BPF_STX | BPF_MEM | BPF_H) ||
kernel/bpf/verifier.c static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 17649
Quoted 8 lines
kernel/bpf/verifier.c +5
-0@@ -17562,6 +17642,11 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)*/ case PTR_TO_BTF_ID |MEM_ALLOC |PTR_UNTRUSTED :if (type ==BPF_READ ) { +/* it is hard to differentiate that the +* BPF_PROBE_MEM is for BPF_MEM or BPF_MEMSX, +* let us use insn->imm to remember it. +*/ + insn->imm =BPF_MODE (insn->code);
That's a fragile approach.
And the evidence is in this patch.
This part of interpreter:
LDX_PROBE_MEM_##SIZEOP: \
bpf_probe_read_kernel(&DST, sizeof(SIZE), \
(const void *)(long) (SRC + insn->off)); \
DST = *((SIZE *)&DST); \wasn't updated to handle sign extension.
How about
#define BPF_PROBE_MEMSX 0x40 /* same as BPF_IND */
and handle it in JITs and interpreter.
We need a selftest for BTF style access to signed fields to make sure both interpreter and JIT handling of BPF_PROBE_MEMSX is tested.
Re: [PATCH bpf-next v2 01/15] bpf: Support new sign-extension load insns
Quoted 9 lines · Alexei Starovoitov
kernel/bpf/core.c static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) · line 1948
Quoted 6 lines · Yonghong Song
kernel/bpf/core.c +1
-0@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \LDSX ?
Ack.
Quoted 37 lines
Quoted 9 lines
kernel/bpf/core.c +8
-0+ LDX_MEMSX_##SIZEOP: \ +DST = *(SIZE *)(unsigned long ) (SRC + insn->off); \ +CONT ; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS...
Quoted 13 lines
kernel/bpf/verifier.c +4
-1@@ -17503,7 +17580,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)if (insn->code == (BPF_LDX |BPF_MEM |BPF_B ) || insn->code == (BPF_LDX |BPF_MEM |BPF_H ) || insn->code == (BPF_LDX |BPF_MEM |BPF_W ) || - insn->code == (BPF_LDX |BPF_MEM |BPF_DW )) { + insn->code == (BPF_LDX |BPF_MEM |BPF_DW ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_B ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_H ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_W )) { type =BPF_READ ; } else if (insn->code == (BPF_STX | BPF_MEM | BPF_B) || insn->code == (BPF_STX | BPF_MEM | BPF_H) ||kernel/bpf/verifier.c static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 17649
Quoted 8 lines
kernel/bpf/verifier.c +5
-0@@ -17562,6 +17642,11 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)*/ case PTR_TO_BTF_ID |MEM_ALLOC |PTR_UNTRUSTED :if (type ==BPF_READ ) { +/* it is hard to differentiate that the +* BPF_PROBE_MEM is for BPF_MEM or BPF_MEMSX, +* let us use insn->imm to remember it. +*/ + insn->imm =BPF_MODE (insn->code);That's a fragile approach.
And the evidence is in this patch.
This part of interpreter:
Quoted 6 lines
LDX_PROBE_MEM_##SIZEOP: \ bpf_probe_read_kernel(&DST, sizeof(SIZE), \ (const void *)(long) (SRC + insn->off)); \ DST = *((SIZE *)&DST); \wasn't updated to handle sign extension.
Thanks for catching this!
Quoted 5 lines
How about
#defineBPF_PROBE_MEMSX0x40/* same asBPF_IND*/and handle it in JITs and interpreter.
Good idea. Will do.
Quoted 2 lines
We need a selftest for BTF style access to signed fields to make sure both interpreter and JIT handling of
BPF_PROBE_MEMSXis tested.
Will do.
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 01/15] bpf: Support new sign-extension load insns
In reply to Yonghong SongQuoted 354 lines · Yonghong Song
Quoted 354 lines
Add interpreter/jit support for new sign-extension load insns which adds a new mode (
BPF_MEMSX).
Also add verifier support to recognize these insns and to do proper verification with new insns. In verifier, besides to deduce proper bounds for thedst_reg, probed memory access is handled by remembering insn mode ininsn->immfield so later on proper jit insns can be emitted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 32 ++++++++- include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 13 ++++ kernel/bpf/verifier.c | 125 +++++++++++++++++++++++++++------ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 151 insertions(+), 21 deletions(-)arch/x86/net/bpf_jit_comp.c +31
-1@@ -779,6 +779,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* LDX: dst_reg = *(s8*)(src_reg + off) */ +static void emit_ldsx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; + +switch (size) { +case BPF_B : +/* Emit 'movsx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBE ); +break ; +case BPF_H : +/* Emit 'movsx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBF ); +break ; +case BPF_W : +/* Emit 'movsx rax, dword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x63 ); +break ; + } +emit_insn_suffix (&prog, src_reg, dst_reg, off); + *pprog = prog; +} +/* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) {@@ -1370,6 +1393,9 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : +case BPF_LDX |BPF_MEMSX |BPF_B : +case BPF_LDX |BPF_MEMSX |BPF_H : +case BPF_LDX |BPF_MEMSX |BPF_W : insn_off = insn->off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {@@ -1415,7 +1441,11 @@ st: if (is_imm8(insn->off)) start_of_ldx = prog; end_of_jmp[-1 ] = start_of_ldx - end_of_jmp; } -emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +if ((BPF_MODE (insn->code) ==BPF_PROBE_MEM && insn->imm ==BPF_MEMSX ) || +BPF_MODE (insn->code) ==BPF_MEMSX ) +emit_ldsx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +else +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {struct exception_table_entry *ex;u8 *_insn = image + proglen + (start_of_ldx - temp);include/uapi/linux/bpf.h +1
-0@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */ kernel/bpf/core.c +13
-0@@ -1610,6 +1610,9 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (LDX ,MEM ,H ), \INSN_3 (LDX ,MEM ,W ), \INSN_3 (LDX ,MEM ,DW ), \ +INSN_3 (LDX ,MEMSX ,B ), \ +INSN_3 (LDX ,MEMSX ,H ), \ +INSN_3 (LDX ,MEMSX ,W ), \/* Immediate based. */ \INSN_3 (LD ,IMM ,DW )@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \ + LDX_MEMSX_##SIZEOP: \ + DST = *(SIZE *)(unsigned long) (SRC + insn->off); \ + CONT; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS +#define ATOMIC_ALU_OP (BOP ,KOP ) \ case BOP: \ if (BPF_SIZE(insn->code) == BPF_W) \kernel/bpf/verifier.c +96
-19@@ -5795,6 +5795,77 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size)__reg_combine_64_into_32 (reg); } +static void set_sext64_default_val (struct bpf_reg_state *reg,int size) +{ +if (size ==1 ) { + reg->smin_value = reg->s32_min_value =S8_MIN ; + reg->smax_value = reg->s32_max_value =S8_MAX ; + }else if (size ==2 ) { + reg->smin_value = reg->s32_min_value =S16_MIN ; + reg->smax_value = reg->s32_max_value =S16_MAX ; + }else { +/* size == 4 */ + reg->smin_value = reg->s32_min_value =S32_MIN ; + reg->smax_value = reg->s32_max_value =S32_MAX ; + } + reg->umin_value = reg->u32_min_value =0 ; + reg->umax_value =U64_MAX ; + reg->u32_max_value =U32_MAX ; + reg->var_off = tnum_unknown; +} + +static void coerce_reg_to_size_sx (struct bpf_reg_state *reg,int size) +{ +u64 top_smax_value, top_smin_value; +s64 init_s64_max, init_s64_min, s64_max, s64_min; +u64 num_bits = size *8 ; + + top_smax_value = ((u64 )reg->smax_value >> num_bits) << num_bits; + top_smin_value = ((u64 )reg->smin_value >> num_bits) << num_bits; + +if (top_smax_value != top_smin_value) + goto out; + +/* find the s64_min and s64_min after sign extension */ +if (size ==1 ) { + init_s64_max = (s8 )reg->smax_value; + init_s64_min = (s8 )reg->smin_value; + }else if (size ==2 ) { + init_s64_max = (s16 )reg->smax_value; + init_s64_min = (s16 )reg->smin_value; + }else { +/* size == 4 */ + init_s64_max = (s32 )reg->smax_value; + init_s64_min = (s32 )reg->smin_value; + } + + s64_max =max (init_s64_max, init_s64_min); + s64_min =min (init_s64_max, init_s64_min); + +if (s64_max >=0 && s64_min >=0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = reg->u32_min_value = s64_min; + reg->umax_value = reg->u32_max_value = s64_max; + reg->var_off =tnum_range (s64_min, s64_max); +return ; + } + +if (s64_min <0 && s64_max <0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = (u64 )s64_max; + reg->umax_value = (u64 )s64_min; + reg->u32_min_value = (u32 )s64_max; + reg->u32_max_value = (u32 )s64_min; + reg->var_off =tnum_range ((u64 )s64_max, (u64 )s64_min); +return ; + } + +out: +set_sext64_default_val (reg, size); +} +static bool bpf_map_is_rdonly (const struct bpf_map *map) {/* A map is considered read-only if the following condition are true: @@ -5815,7 +5886,8 @@ static bool bpf_map_is_rdonly(const struct bpf_map *map) !bpf_map_write_active (map); } -static int bpf_map_direct_read (struct bpf_map *map,int off,int size,u64 *val) +static int bpf_map_direct_read (struct bpf_map *map,int off,int size,u64 *val, +bool is_ldsx) {void *ptr;u64 addr;@@ -5828,13 +5900,13 @@ static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val)switch (size) {case sizeof (u8): - *val = (u64)*(u8 *)ptr; + *val = is_ldsx ? (s64)*(s8 *)ptr : (u64)*(u8 *)ptr;break ;case sizeof (u16): - *val = (u64)*(u16 *)ptr; + *val = is_ldsx ? (s64)*(s16 *)ptr : (u64)*(u16 *)ptr;break ;case sizeof (u32): - *val = (u64)*(u32 *)ptr; + *val = is_ldsx ? (s64)*(s32 *)ptr : (u64)*(u32 *)ptr;break ;case sizeof (u64): *val = *(u64 *)ptr;@@ -6248,7 +6320,7 @@ static int check_stack_access_within_bounds(*/ static int check_mem_access (struct bpf_verifier_env *env,int insn_idx,u32 regno,int off,int bpf_size,enum bpf_access_type t, -int value_regno,bool strict_alignment_once) +int value_regno,bool strict_alignment_once,bool is_ldsx) {struct bpf_reg_state *regs =cur_regs (env);struct bpf_reg_state *reg = regs + regno;@@ -6309,7 +6381,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnu64 val =0 ; err =bpf_map_direct_read (map, map_off, size, - &val); + &val, is_ldsx);if (err)return err;@@ -6479,8 +6551,11 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (!err && size <BPF_REG_SIZE && value_regno >=0 && t ==BPF_READ && regs[value_regno].type ==SCALAR_VALUE ) { -/* b/h/w load zero-extends, mark upper bits as known 0 */ -coerce_reg_to_size (®s[value_regno], size); +if (!is_ldsx) +/* b/h/w load zero-extends, mark upper bits as known 0 */ +coerce_reg_to_size (®s[value_regno], size); +else +coerce_reg_to_size_sx (®s[value_regno], size); }return err; }@@ -6572,17 +6647,17 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i* case to simulate the register fill. */ err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off, -BPF_SIZE (insn->code),BPF_READ ,-1 , true); +BPF_SIZE (insn->code),BPF_READ ,-1 , true, false);if (!err && load_reg >=0 ) err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_READ , load_reg, - true); + true, false);if (err)return err;/* Check whether we can write into the same memory. */ err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off, -BPF_SIZE (insn->code),BPF_WRITE ,-1 , true); +BPF_SIZE (insn->code),BPF_WRITE ,-1 , true, false);if (err)return err;@@ -6828,7 +6903,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,return zero_size_allowed ?0 : -EACCES ;return check_mem_access (env, env->insn_idx, regno, offset,BPF_B , - atype,-1 , false); + atype,-1 , false, false); } fallthrough;@@ -7200,7 +7275,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn/* we write BPF_DW bits (8 bytes) at a time */ for (i =0 ; i <BPF_DYNPTR_SIZE ; i +=8 ) { err =check_mem_access (env, insn_idx, regno, - i,BPF_DW ,BPF_WRITE ,-1 , false); + i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -7293,7 +7368,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idfor (i =0 ; i < nr_slots *8 ; i +=BPF_REG_SIZE ) { err =check_mem_access (env, insn_idx, regno, - i,BPF_DW ,BPF_WRITE ,-1 , false); + i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -9437,7 +9512,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn*/ for (i =0 ; i < meta.access_size; i++) { err =check_mem_access (env, insn_idx, meta.regno, i,BPF_B , -BPF_WRITE ,-1 , false); +BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -16315,7 +16390,8 @@ static int do_check(struct bpf_verifier_env *env)*/ err =check_mem_access (env, env->insn_idx, insn->src_reg, insn->off,BPF_SIZE (insn->code), -BPF_READ , insn->dst_reg, false); +BPF_READ , insn->dst_reg, false, +BPF_MODE (insn->code) ==BPF_MEMSX );if (err)return err;@@ -16352,7 +16428,7 @@ static int do_check(struct bpf_verifier_env *env)/* check that memory (dst_reg + off) is writeable */ err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code), -BPF_WRITE , insn->src_reg, false); +BPF_WRITE , insn->src_reg, false, false);if (err)return err;@@ -16377,7 +16453,7 @@ static int do_check(struct bpf_verifier_env *env)/* check that memory (dst_reg + off) is writeable */ err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code), -BPF_WRITE ,-1 , false); +BPF_WRITE ,-1 , false, false);if (err)return err;@@ -16805,7 +16881,8 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)for (i =0 ; i < insn_cnt; i++, insn++) {if (BPF_CLASS (insn->code) ==BPF_LDX && - (BPF_MODE (insn->code) !=BPF_MEM || insn->imm !=0 )) { + ((BPF_MODE (insn->code) !=BPF_MEM &&BPF_MODE (insn->code) !=BPF_MEMSX ) || + insn->imm !=0 )) {verbose (env,"BPF_LDX uses reserved fields\n" );return -EINVAL ; }@@ -17503,7 +17580,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
Quoted 8 lines
kernel/bpf/verifier.c static int convert_ctx_accesses(struct bpf_verifier_env *env) · line 17586
Quoted 8 lines
kernel/bpf/verifier.c +4
-1if (insn->code == (BPF_LDX |BPF_MEM |BPF_B ) || insn->code == (BPF_LDX |BPF_MEM |BPF_H ) || insn->code == (BPF_LDX |BPF_MEM |BPF_W ) || - insn->code == (BPF_LDX |BPF_MEM |BPF_DW )) { + insn->code == (BPF_LDX |BPF_MEM |BPF_DW ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_B ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_H ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_W )) {
Later in this function there is a code that deals with is_narrower_load condition (line 17785 in my case).
This code handles the case when e.g. 1 byte is read from a 4 byte field. It does so by first converting such load to 4 byte load and than adding BPF_RSH and BPF_AND instructions.
It appears to me that this code should handle sign extension as well.
Quoted 27 lines
Quoted 27 lines
kernel/bpf/verifier.c +5
-0type =BPF_READ ; } else if (insn->code == (BPF_STX | BPF_MEM | BPF_B) || insn->code == (BPF_STX | BPF_MEM | BPF_H) ||@@ -17562,6 +17642,11 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)*/ case PTR_TO_BTF_ID |MEM_ALLOC |PTR_UNTRUSTED :if (type ==BPF_READ ) { +/* it is hard to differentiate that the +* BPF_PROBE_MEM is for BPF_MEM or BPF_MEMSX, +* let us use insn->imm to remember it. +*/ + insn->imm =BPF_MODE (insn->code); insn->code =BPF_LDX |BPF_PROBE_MEM |BPF_SIZE ((insn)->code); env->prog->aux->num_exentries++;tools/include/uapi/linux/bpf.h +1
-0@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 01/15] bpf: Support new sign-extension load insns
In reply to Yonghong SongQuoted 176 lines · Yonghong Song
Add interpreter/jit support for new sign-extension load insns which adds a new mode (
BPF_MEMSX).
Also add verifier support to recognize these insns and to do proper verification with new insns. In verifier, besides to deduce proper bounds for thedst_reg, probed memory access is handled by remembering insn mode ininsn->immfield so later on proper jit insns can be emitted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 32 ++++++++- include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 13 ++++ kernel/bpf/verifier.c | 125 +++++++++++++++++++++++++++------ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 151 insertions(+), 21 deletions(-)arch/x86/net/bpf_jit_comp.c +31
-1@@ -779,6 +779,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* LDX: dst_reg = *(s8*)(src_reg + off) */ +static void emit_ldsx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; + +switch (size) { +case BPF_B : +/* Emit 'movsx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBE ); +break ; +case BPF_H : +/* Emit 'movsx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBF ); +break ; +case BPF_W : +/* Emit 'movsx rax, dword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x63 ); +break ; + } +emit_insn_suffix (&prog, src_reg, dst_reg, off); + *pprog = prog; +} +/* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) {@@ -1370,6 +1393,9 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : +case BPF_LDX |BPF_MEMSX |BPF_B : +case BPF_LDX |BPF_MEMSX |BPF_H : +case BPF_LDX |BPF_MEMSX |BPF_W : insn_off = insn->off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {@@ -1415,7 +1441,11 @@ st: if (is_imm8(insn->off)) start_of_ldx = prog; end_of_jmp[-1 ] = start_of_ldx - end_of_jmp; } -emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +if ((BPF_MODE (insn->code) ==BPF_PROBE_MEM && insn->imm ==BPF_MEMSX ) || +BPF_MODE (insn->code) ==BPF_MEMSX ) +emit_ldsx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +else +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {struct exception_table_entry *ex;u8 *_insn = image + proglen + (start_of_ldx - temp);include/uapi/linux/bpf.h +1
-0@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */ kernel/bpf/core.c +13
-0@@ -1610,6 +1610,9 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (LDX ,MEM ,H ), \INSN_3 (LDX ,MEM ,W ), \INSN_3 (LDX ,MEM ,DW ), \ +INSN_3 (LDX ,MEMSX ,B ), \ +INSN_3 (LDX ,MEMSX ,H ), \ +INSN_3 (LDX ,MEMSX ,W ), \/* Immediate based. */ \INSN_3 (LD ,IMM ,DW )@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \ + LDX_MEMSX_##SIZEOP: \ + DST = *(SIZE *)(unsigned long) (SRC + insn->off); \ + CONT; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS +#define ATOMIC_ALU_OP (BOP ,KOP ) \ case BOP: \ if (BPF_SIZE(insn->code) == BPF_W) \kernel/bpf/verifier.c +51
-0@@ -5795,6 +5795,77 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size)__reg_combine_64_into_32 (reg); } +static void set_sext64_default_val (struct bpf_reg_state *reg,int size) +{ +if (size ==1 ) { + reg->smin_value = reg->s32_min_value =S8_MIN ; + reg->smax_value = reg->s32_max_value =S8_MAX ; + }else if (size ==2 ) { + reg->smin_value = reg->s32_min_value =S16_MIN ; + reg->smax_value = reg->s32_max_value =S16_MAX ; + }else { +/* size == 4 */ + reg->smin_value = reg->s32_min_value =S32_MIN ; + reg->smax_value = reg->s32_max_value =S32_MAX ; + } + reg->umin_value = reg->u32_min_value =0 ; + reg->umax_value =U64_MAX ; + reg->u32_max_value =U32_MAX ; + reg->var_off = tnum_unknown; +} + +static void coerce_reg_to_size_sx (struct bpf_reg_state *reg,int size) +{ +u64 top_smax_value, top_smin_value; +s64 init_s64_max, init_s64_min, s64_max, s64_min; +u64 num_bits = size *8 ; + + top_smax_value = ((u64 )reg->smax_value >> num_bits) << num_bits; + top_smin_value = ((u64 )reg->smin_value >> num_bits) << num_bits; + +if (top_smax_value != top_smin_value) + goto out; + +/* find the s64_min and s64_min after sign extension */ +if (size ==1 ) { + init_s64_max = (s8 )reg->smax_value; + init_s64_min = (s8 )reg->smin_value; + }else if (size ==2 ) { + init_s64_max = (s16 )reg->smax_value; + init_s64_min = (s16 )reg->smin_value; + }else { +/* size == 4 */ + init_s64_max = (s32 )reg->smax_value; + init_s64_min = (s32 )reg->smin_value; + } + + s64_max =max (init_s64_max, init_s64_min); + s64_min =min (init_s64_max, init_s64_min); + +if (s64_max >=0 && s64_min >=0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = reg->u32_min_value = s64_min;
kernel/bpf/verifier.c static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · line 5858
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ reg->umax_value = reg->u32_max_value = s64_max; + reg->var_off =tnum_range (s64_min, s64_max); +return ; + } + +if (s64_min <0 && s64_max <0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = (u64 )s64_max; + reg->umax_value = (u64 )s64_min;
I think the last two assignments are not correct for the following example:
{
"testtesttest",
.insns = {
BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32),
BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 0xff80, 2),
BPF_JMP_IMM(BPF_JGT, BPF_REG_0, 0xffff, 1),
{
.code = BPF_ALU64 | BPF_MOV | BPF_X,
.dst_reg = BPF_REG_0,
.src_reg = BPF_REG_0,
.off = 8,
.imm = 0,
},
BPF_EXIT_INSN(),
},
.result = ACCEPT,
.retval = 0,
},Here is execution log:
0: R1=ctx(off=0,imm=0) R10=fp0
0: (85) call bpf_get_prandom_u32#7 ; R0_w=Pscalar()
1: (a5) if r0 < 0xff80 goto pc+2 ; R0_w=Pscalar(umin=65408)
2: (25) if r0 > 0xffff goto pc+1 ; R0_w=Pscalar(umin=65408,umax=65535,var_off=(0xff80; 0x7f))
3: (bf) r0 = r0 ; R0_w=Pscalar
(smin=-128,smax=-1,
umin=18'446'744'073'709'551'615,
umax=18'446'744'073'709'551'488,
var_off=(0xffffffffffffff80; 0x7f),
u32_min=-1,u32_max=-128)4: (95) exit
Note that umax < umin, which should not happen.
In this case the assignments in question are:
reg->umin_value = (u64)s64_max; // == -1 == 0xffffffffffffffff
reg->umax_value = (u64)s64_min; // == -128 == 0xffffffffffffff80Quoted 203 lines
kernel/bpf/verifier.c +44
-20+ reg->u32_min_value = (u32 )s64_max; + reg->u32_max_value = (u32 )s64_min; + reg->var_off =tnum_range ((u64 )s64_max, (u64 )s64_min); +return ; + } + +out: +set_sext64_default_val (reg, size); +} +static bool bpf_map_is_rdonly (const struct bpf_map *map) {/* A map is considered read-only if the following condition are true: @@ -5815,7 +5886,8 @@ static bool bpf_map_is_rdonly(const struct bpf_map *map) !bpf_map_write_active (map); } -static int bpf_map_direct_read (struct bpf_map *map,int off,int size,u64 *val) +static int bpf_map_direct_read (struct bpf_map *map,int off,int size,u64 *val, +bool is_ldsx) {void *ptr;u64 addr;@@ -5828,13 +5900,13 @@ static int bpf_map_direct_read(struct bpf_map *map, int off, int size, u64 *val)switch (size) {case sizeof (u8): - *val = (u64)*(u8 *)ptr; + *val = is_ldsx ? (s64)*(s8 *)ptr : (u64)*(u8 *)ptr;break ;case sizeof (u16): - *val = (u64)*(u16 *)ptr; + *val = is_ldsx ? (s64)*(s16 *)ptr : (u64)*(u16 *)ptr;break ;case sizeof (u32): - *val = (u64)*(u32 *)ptr; + *val = is_ldsx ? (s64)*(s32 *)ptr : (u64)*(u32 *)ptr;break ;case sizeof (u64): *val = *(u64 *)ptr;@@ -6248,7 +6320,7 @@ static int check_stack_access_within_bounds(*/ static int check_mem_access (struct bpf_verifier_env *env,int insn_idx,u32 regno,int off,int bpf_size,enum bpf_access_type t, -int value_regno,bool strict_alignment_once) +int value_regno,bool strict_alignment_once,bool is_ldsx) {struct bpf_reg_state *regs =cur_regs (env);struct bpf_reg_state *reg = regs + regno;@@ -6309,7 +6381,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnu64 val =0 ; err =bpf_map_direct_read (map, map_off, size, - &val); + &val, is_ldsx);if (err)return err;@@ -6479,8 +6551,11 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regnif (!err && size <BPF_REG_SIZE && value_regno >=0 && t ==BPF_READ && regs[value_regno].type ==SCALAR_VALUE ) { -/* b/h/w load zero-extends, mark upper bits as known 0 */ -coerce_reg_to_size (®s[value_regno], size); +if (!is_ldsx) +/* b/h/w load zero-extends, mark upper bits as known 0 */ +coerce_reg_to_size (®s[value_regno], size); +else +coerce_reg_to_size_sx (®s[value_regno], size); }return err; }@@ -6572,17 +6647,17 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i* case to simulate the register fill. */ err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off, -BPF_SIZE (insn->code),BPF_READ ,-1 , true); +BPF_SIZE (insn->code),BPF_READ ,-1 , true, false);if (!err && load_reg >=0 ) err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code),BPF_READ , load_reg, - true); + true, false);if (err)return err;/* Check whether we can write into the same memory. */ err =check_mem_access (env, insn_idx, insn->dst_reg, insn->off, -BPF_SIZE (insn->code),BPF_WRITE ,-1 , true); +BPF_SIZE (insn->code),BPF_WRITE ,-1 , true, false);if (err)return err;@@ -6828,7 +6903,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,return zero_size_allowed ?0 : -EACCES ;return check_mem_access (env, env->insn_idx, regno, offset,BPF_B , - atype,-1 , false); + atype,-1 , false, false); } fallthrough;@@ -7200,7 +7275,7 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn/* we write BPF_DW bits (8 bytes) at a time */ for (i =0 ; i <BPF_DYNPTR_SIZE ; i +=8 ) { err =check_mem_access (env, insn_idx, regno, - i,BPF_DW ,BPF_WRITE ,-1 , false); + i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -7293,7 +7368,7 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idfor (i =0 ; i < nr_slots *8 ; i +=BPF_REG_SIZE ) { err =check_mem_access (env, insn_idx, regno, - i,BPF_DW ,BPF_WRITE ,-1 , false); + i,BPF_DW ,BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -9437,7 +9512,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn*/ for (i =0 ; i < meta.access_size; i++) { err =check_mem_access (env, insn_idx, meta.regno, i,BPF_B , -BPF_WRITE ,-1 , false); +BPF_WRITE ,-1 , false, false);if (err)return err; }@@ -16315,7 +16390,8 @@ static int do_check(struct bpf_verifier_env *env)*/ err =check_mem_access (env, env->insn_idx, insn->src_reg, insn->off,BPF_SIZE (insn->code), -BPF_READ , insn->dst_reg, false); +BPF_READ , insn->dst_reg, false, +BPF_MODE (insn->code) ==BPF_MEMSX );if (err)return err;@@ -16352,7 +16428,7 @@ static int do_check(struct bpf_verifier_env *env)/* check that memory (dst_reg + off) is writeable */ err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code), -BPF_WRITE , insn->src_reg, false); +BPF_WRITE , insn->src_reg, false, false);if (err)return err;@@ -16377,7 +16453,7 @@ static int do_check(struct bpf_verifier_env *env)/* check that memory (dst_reg + off) is writeable */ err =check_mem_access (env, env->insn_idx, insn->dst_reg, insn->off,BPF_SIZE (insn->code), -BPF_WRITE ,-1 , false); +BPF_WRITE ,-1 , false, false);if (err)return err;@@ -16805,7 +16881,8 @@ static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)for (i =0 ; i < insn_cnt; i++, insn++) {if (BPF_CLASS (insn->code) ==BPF_LDX && - (BPF_MODE (insn->code) !=BPF_MEM || insn->imm !=0 )) { + ((BPF_MODE (insn->code) !=BPF_MEM &&BPF_MODE (insn->code) !=BPF_MEMSX ) || + insn->imm !=0 )) {verbose (env,"BPF_LDX uses reserved fields\n" );return -EINVAL ; }@@ -17503,7 +17580,10 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)if (insn->code == (BPF_LDX |BPF_MEM |BPF_B ) || insn->code == (BPF_LDX |BPF_MEM |BPF_H ) || insn->code == (BPF_LDX |BPF_MEM |BPF_W ) || - insn->code == (BPF_LDX |BPF_MEM |BPF_DW )) { + insn->code == (BPF_LDX |BPF_MEM |BPF_DW ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_B ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_H ) || + insn->code == (BPF_LDX |BPF_MEMSX |BPF_W )) { type =BPF_READ ; } else if (insn->code == (BPF_STX | BPF_MEM | BPF_B) || insn->code == (BPF_STX | BPF_MEM | BPF_H) ||@@ -17562,6 +17642,11 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)*/ case PTR_TO_BTF_ID |MEM_ALLOC |PTR_UNTRUSTED :if (type ==BPF_READ ) { +/* it is hard to differentiate that the +* BPF_PROBE_MEM is for BPF_MEM or BPF_MEMSX, +* let us use insn->imm to remember it. +*/ + insn->imm =BPF_MODE (insn->code); insn->code =BPF_LDX |BPF_PROBE_MEM |BPF_SIZE ((insn)->code); env->prog->aux->num_exentries++;tools/include/uapi/linux/bpf.h +1
-0@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */
Re: [PATCH bpf-next v2 01/15] bpf: Support new sign-extension load insns
Quoted 221 lines · Eduard Zingerman
Quoted 176 lines · Yonghong Song
Add interpreter/jit support for new sign-extension load insns which adds a new mode (
BPF_MEMSX).
Also add verifier support to recognize these insns and to do proper verification with new insns. In verifier, besides to deduce proper bounds for thedst_reg, probed memory access is handled by remembering insn mode ininsn->immfield so later on proper jit insns can be emitted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 32 ++++++++- include/uapi/linux/bpf.h | 1 + kernel/bpf/core.c | 13 ++++ kernel/bpf/verifier.c | 125 +++++++++++++++++++++++++++------ tools/include/uapi/linux/bpf.h | 1 + 5 files changed, 151 insertions(+), 21 deletions(-)arch/x86/net/bpf_jit_comp.c +31
-1@@ -779,6 +779,29 @@ static void emit_ldx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } +/* LDX: dst_reg = *(s8*)(src_reg + off) */ +static void emit_ldsx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) +{ +u8 *prog = *pprog; + +switch (size) { +case BPF_B : +/* Emit 'movsx rax, byte ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBE ); +break ; +case BPF_H : +/* Emit 'movsx rax, word ptr [rax + off]' */ +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x0F ,0xBF ); +break ; +case BPF_W : +/* Emit 'movsx rax, dword ptr [rax+0x14]' */ +EMIT2 (add_2mod (0x48 , src_reg, dst_reg),0x63 ); +break ; + } +emit_insn_suffix (&prog, src_reg, dst_reg, off); + *pprog = prog; +} +/* STX: *(u8*)(dst_reg + off) = src_reg */ static void emit_stx (u8 **pprog,u32 size,u32 dst_reg,u32 src_reg,int off) {@@ -1370,6 +1393,9 @@ st: if (is_imm8(insn->off))case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_DW : +case BPF_LDX |BPF_MEMSX |BPF_B : +case BPF_LDX |BPF_MEMSX |BPF_H : +case BPF_LDX |BPF_MEMSX |BPF_W : insn_off = insn->off;if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {@@ -1415,7 +1441,11 @@ st: if (is_imm8(insn->off)) start_of_ldx = prog; end_of_jmp[-1 ] = start_of_ldx - end_of_jmp; } -emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +if ((BPF_MODE (insn->code) ==BPF_PROBE_MEM && insn->imm ==BPF_MEMSX ) || +BPF_MODE (insn->code) ==BPF_MEMSX ) +emit_ldsx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off); +else +emit_ldx (&prog,BPF_SIZE (insn->code), dst_reg, src_reg, insn_off);if (BPF_MODE (insn->code) ==BPF_PROBE_MEM ) {struct exception_table_entry *ex;u8 *_insn = image + proglen + (start_of_ldx - temp);include/uapi/linux/bpf.h +1
-0@@ -19,6 +19,7 @@/* ld/ldx fields */ #define BPF_DW 0x18/* double word (64-bit) */ +#define BPF_MEMSX 0x80/* load with sign extension */ #define BPF_ATOMIC 0xc0/* atomic memory ops - op type in immediate */ #define BPF_XADD 0xc0/* exclusive add - legacy name */ kernel/bpf/core.c +13
-0@@ -1610,6 +1610,9 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (LDX ,MEM ,H ), \INSN_3 (LDX ,MEM ,W ), \INSN_3 (LDX ,MEM ,DW ), \ +INSN_3 (LDX ,MEMSX ,B ), \ +INSN_3 (LDX ,MEMSX ,H ), \ +INSN_3 (LDX ,MEMSX ,W ), \/* Immediate based. */ \INSN_3 (LD ,IMM ,DW )@@ -1942,6 +1945,16 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) LDST(DW, u64)#undef LDST +#define LDS (SIZEOP ,SIZE ) \ + LDX_MEMSX_##SIZEOP: \ + DST = *(SIZE *)(unsigned long) (SRC + insn->off); \ + CONT; + + LDS(B, s8) + LDS(H, s16) + LDS(W, s32) +#undef LDS +#define ATOMIC_ALU_OP (BOP ,KOP ) \ case BOP: \ if (BPF_SIZE(insn->code) == BPF_W) \kernel/bpf/verifier.c +51
-0@@ -5795,6 +5795,77 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size)__reg_combine_64_into_32 (reg); } +static void set_sext64_default_val (struct bpf_reg_state *reg,int size) +{ +if (size ==1 ) { + reg->smin_value = reg->s32_min_value =S8_MIN ; + reg->smax_value = reg->s32_max_value =S8_MAX ; + }else if (size ==2 ) { + reg->smin_value = reg->s32_min_value =S16_MIN ; + reg->smax_value = reg->s32_max_value =S16_MAX ; + }else { +/* size == 4 */ + reg->smin_value = reg->s32_min_value =S32_MIN ; + reg->smax_value = reg->s32_max_value =S32_MAX ; + } + reg->umin_value = reg->u32_min_value =0 ; + reg->umax_value =U64_MAX ; + reg->u32_max_value =U32_MAX ; + reg->var_off = tnum_unknown; +} + +static void coerce_reg_to_size_sx (struct bpf_reg_state *reg,int size) +{ +u64 top_smax_value, top_smin_value; +s64 init_s64_max, init_s64_min, s64_max, s64_min; +u64 num_bits = size *8 ; + + top_smax_value = ((u64 )reg->smax_value >> num_bits) << num_bits; + top_smin_value = ((u64 )reg->smin_value >> num_bits) << num_bits; + +if (top_smax_value != top_smin_value) + goto out; + +/* find the s64_min and s64_min after sign extension */ +if (size ==1 ) { + init_s64_max = (s8 )reg->smax_value; + init_s64_min = (s8 )reg->smin_value; + }else if (size ==2 ) { + init_s64_max = (s16 )reg->smax_value; + init_s64_min = (s16 )reg->smin_value; + }else { +/* size == 4 */ + init_s64_max = (s32 )reg->smax_value; + init_s64_min = (s32 )reg->smin_value; + } + + s64_max =max (init_s64_max, init_s64_min); + s64_min =min (init_s64_max, init_s64_min); + +if (s64_max >=0 && s64_min >=0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = reg->u32_min_value = s64_min;kernel/bpf/verifier.c static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · line 5858
Quoted 10 lines
kernel/bpf/verifier.c +10
-0+ reg->umax_value = reg->u32_max_value = s64_max; + reg->var_off =tnum_range (s64_min, s64_max); +return ; + } + +if (s64_min <0 && s64_max <0 ) { + reg->smin_value = reg->s32_min_value = s64_min; + reg->smax_value = reg->s32_max_value = s64_max; + reg->umin_value = (u64 )s64_max; + reg->umax_value = (u64 )s64_min;I think the last two assignments are not correct for the following example:
{ "testtesttest", .insns = { BPF_EMIT_CALL(BPF_FUNC_get_prandom_u32), BPF_JMP_IMM(BPF_JLT, BPF_REG_0, 0xff80, 2), BPF_JMP_IMM(BPF_JGT, BPF_REG_0, 0xffff, 1), { .code = BPF_ALU64 | BPF_MOV | BPF_X, .dst_reg = BPF_REG_0, .src_reg = BPF_REG_0, .off = 8, .imm = 0, }, BPF_EXIT_INSN(), }, .result = ACCEPT, .retval = 0, },Here is execution log:
0: R1=ctx(off=0,imm=0) R10=fp0
0: (85) call bpf_get_prandom_u32#7 ; R0_w=Pscalar()1: (a5) if r0 < 0xff80 goto pc+2 ; R0_w=Pscalar(umin=65408) 2: (25) if r0 > 0xffff goto pc+1 ; R0_w=Pscalar(umin=65408,umax=65535,var_off=(0xff80; 0x7f)) 3: (bf) r0 = r0 ; R0_w=Pscalar (smin=-128,smax=-1, umin=18'446'744'073'709'551'615, umax=18'446'744'073'709'551'488, var_off=(0xffffffffffffff80; 0x7f), u32_min=-1,u32_max=-128)
Quoted 7 lines
4: (95) exit
Note that umax < umin, which should not happen.
In this case the assignments in question are:reg->umin_value = (u64)s64_max; // == -1 == 0xffffffffffffffff reg->umax_value = (u64)s64_min; // == -128 == 0xffffffffffffff80
Thanks for pointing out. Yes, the assignment is incorrect and they are mismatched. Will fix the issue and add a test for this.
Quoted 13 lines
Quoted 12 lines
kernel/bpf/verifier.c +10
-0+ reg->u32_min_value = (u32 )s64_max; + reg->u32_max_value = (u32 )s64_min; + reg->var_off =tnum_range ((u64 )s64_max, (u64 )s64_min); +return ; + } + +out: +set_sext64_default_val (reg, size); +} +[...]
PATCH v2 2/15 bpf: Fix sign-extension ctx member accesses
In reply to Yonghong SongIn uapi bpf.h, there are two ctx structures which contain signed members. Without cpu v4, such signed members will be loaded with unsigned load and the compiler will generate proper left and right shifts to get the proper final value.
With sign-extension load, however, left/right shifts are gone, we need to ensure these signed members are properly handled, with signed loads or other means. The following are list of signed ctx members and how they are handled.
(1).
struct bpf_sock {
...
__s32 rx_queue_mapping;}
The corresponding kernel fields are
struct sock_common {
...
unsigned short skc_rx_queue_mapping;
...}
Current ctx rewriter uses unsigned load for the kernel field which is correct and does not need further handling.
(2).
struct bpf_sockopt {
...
__s32 level;
__s32 optname;
__s32 optlen;
__s32 retval;
}
The level/optname/optlen are from struct bpf_sockopt_kern
struct bpf_sockopt_kern {
...
s32 level;
s32 optname;
s32 optlen;
...
}
and the 'retval' is from struct bpf_cg_run_ctx
struct bpf_cg_run_ctx {
...
int retval; }
Current the above four fields are loaded with unsigned load. Let us modify the read macro for bpf_sockopt which use the same signedness for the original insn.
--- kernel/bpf/cgroup.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
kernel/bpf/cgroup.c +8 -6
@@ -2397,9 +2397,10 @@ static bool cg_sockopt_is_valid_access(int off, int size, }#define CG_SOCKOPT_READ_FIELD (F ) \ - BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F), \ + BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \ + BPF_MODE(si->code) | BPF_CLASS(si->code)), \ si->dst_reg, si->src_reg, \ - offsetof(struct bpf_sockopt_kern, F)) + offsetof(struct bpf_sockopt_kern, F), si->imm)#define CG_SOCKOPT_WRITE_FIELD (F ) \ BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \
@@ -2456,7 +2457,7 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), treg, treg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MEM | + *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MODE(si->code) | BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), treg, si->src_reg, offsetof(struct bpf_cg_run_ctx, retval),
@@ -2470,9 +2471,10 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), si->dst_reg, si->dst_reg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), - si->dst_reg, si->dst_reg, - offsetof(struct bpf_cg_run_ctx , retval)); + *insn++ = BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval) | +BPF_MODE (si->code) |BPF_CLASS (si->code)), + si->dst_reg, si->dst_reg, + offsetof(struct bpf_cg_run_ctx , retval), si->imm); }break ;case offsetof(struct bpf_sockopt , optval):
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 02/15] bpf: Fix sign-extension ctx member accesses
Quoted 9 lines · Yonghong Song
Quoted 9 lines
In uapi
bpf.h, there are two ctx structures which contain signed members. Without cpu v4, such signed members will be loaded with unsigned load and the compiler will generate proper left and right shifts to get the proper final value.With sign-extension load, however, left/right shifts are gone, we need to ensure these signed members are properly handled, with signed loads or other means. The following are list of signed ctx members and how they are handled.
This is not a generic approach, in theory any field could be cast as a signed integer. Do we want to support this? If so, then it should be handled in convert_ctx_access() by generating additional sign extension instructions.
Quoted 87 lines
Quoted 87 lines
(1).
struct bpf_sock { ... __s32 rx_queue_mapping;}
The corresponding kernel fields are
struct sock_common { ... unsigned short skc_rx_queue_mapping; ...}
Current ctx rewriter uses unsigned load for the kernel field which is correct and does not need further handling.
(2).
struct bpf_sockopt { ... __s32 level; __s32 optname; __s32 optlen; __s32 retval; } The level/optname/optlen are from struct bpf_sockopt_kern struct bpf_sockopt_kern { ... s32 level; s32 optname; s32 optlen; ... } and the 'retval' is from struct bpf_cg_run_ctx struct bpf_cg_run_ctx { ... int retval;}
Current the above four fields are loaded with unsigned load. Let us modify the read macro forbpf_sockoptwhich use the same signedness for the original insn.Signed-off-by Yonghong Song <[email protected]>--- kernel/bpf/cgroup.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)kernel/bpf/cgroup.c +8
-6@@ -2397,9 +2397,10 @@ static bool cg_sockopt_is_valid_access(int off, int size, }#define CG_SOCKOPT_READ_FIELD (F ) \ - BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F), \ + BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \ + BPF_MODE(si->code) | BPF_CLASS(si->code)), \ si->dst_reg, si->src_reg, \ - offsetof(struct bpf_sockopt_kern, F)) + offsetof(struct bpf_sockopt_kern, F), si->imm)#define CG_SOCKOPT_WRITE_FIELD (F ) \ BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \@@ -2456,7 +2457,7 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), treg, treg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MEM | + *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MODE(si->code) | BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), treg, si->src_reg, offsetof(struct bpf_cg_run_ctx, retval),@@ -2470,9 +2471,10 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), si->dst_reg, si->dst_reg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), - si->dst_reg, si->dst_reg, - offsetof(struct bpf_cg_run_ctx , retval)); + *insn++ = BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval) | +BPF_MODE (si->code) |BPF_CLASS (si->code)), + si->dst_reg, si->dst_reg, + offsetof(struct bpf_cg_run_ctx , retval), si->imm); }break ;case offsetof(struct bpf_sockopt , optval):
Re: [PATCH bpf-next v2 02/15] bpf: Fix sign-extension ctx member accesses
Quoted 11 lines · Eduard Zingerman
Quoted 9 lines · Yonghong Song
Quoted 9 lines
In uapi
bpf.h, there are two ctx structures which contain signed members. Without cpu v4, such signed members will be loaded with unsigned load and the compiler will generate proper left and right shifts to get the proper final value.With sign-extension load, however, left/right shifts are gone, we need to ensure these signed members are properly handled, with signed loads or other means. The following are list of signed ctx members and how they are handled.
Quoted 4 lines
This is not a generic approach, in theory any field could be cast as a signed integer. Do we want to support this? If so, then it should be handled in
convert_ctx_access()by generating additional sign extension instructions.
Good point. I will make change in verifier.c which is more generic and cover more cases.
Quoted 89 lines
Quoted 88 lines
Quoted 88 lines
(1).
struct bpf_sock { ... __s32 rx_queue_mapping;}
The corresponding kernel fields are
struct sock_common { ... unsigned short skc_rx_queue_mapping; ...}
Current ctx rewriter uses unsigned load for the kernel field which is correct and does not need further handling.
(2).
struct bpf_sockopt { ... __s32 level; __s32 optname; __s32 optlen; __s32 retval; } The level/optname/optlen are from struct bpf_sockopt_kern struct bpf_sockopt_kern { ... s32 level; s32 optname; s32 optlen; ... } and the 'retval' is from struct bpf_cg_run_ctx struct bpf_cg_run_ctx { ... int retval;}
Current the above four fields are loaded with unsigned load. Let us modify the read macro forbpf_sockoptwhich use the same signedness for the original insn.Signed-off-by Yonghong Song <[email protected]>--- kernel/bpf/cgroup.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)kernel/bpf/cgroup.c +8
-6@@ -2397,9 +2397,10 @@ static bool cg_sockopt_is_valid_access(int off, int size, }#define CG_SOCKOPT_READ_FIELD (F ) \ - BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F), \ + BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \ + BPF_MODE(si->code) | BPF_CLASS(si->code)), \ si->dst_reg, si->src_reg, \ - offsetof(struct bpf_sockopt_kern, F)) + offsetof(struct bpf_sockopt_kern, F), si->imm)#define CG_SOCKOPT_WRITE_FIELD (F ) \ BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_sockopt_kern, F) | \@@ -2456,7 +2457,7 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), treg, treg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MEM | + *insn++ = BPF_RAW_INSN(BPF_CLASS(si->code) | BPF_MODE(si->code) | BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), treg, si->src_reg, offsetof(struct bpf_cg_run_ctx, retval),@@ -2470,9 +2471,10 @@ static u32 cg_sockopt_convert_ctx_access(enum bpf_access_type type, *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct task_struct, bpf_ctx), si->dst_reg, si->dst_reg, offsetof(struct task_struct , bpf_ctx)); - *insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval), - si->dst_reg, si->dst_reg, - offsetof(struct bpf_cg_run_ctx , retval)); + *insn++ = BPF_RAW_INSN((BPF_FIELD_SIZEOF(struct bpf_cg_run_ctx, retval) | +BPF_MODE (si->code) |BPF_CLASS (si->code)), + si->dst_reg, si->dst_reg, + offsetof(struct bpf_cg_run_ctx , retval), si->imm); }break ;case offsetof(struct bpf_sockopt , optval):
PATCH v2 3/15 bpf: Support new sign-extension mov insns
In reply to Yonghong SongAdd interpreter/jit support for new sign-extension mov insns. The original 'MOV' insn is extended to support signed version for both ALU and ALU64 operations. For ALU mode, the insn->off value of 8 or 16 indicates sign-extension from 8- or 16-bit value to 32-bit value. For ALU64 mode, the insn->off value of 8/16/32 indicates sign-extension from 8-, 16- or 32-bit value to 64-bit value.
--- arch/x86/net/bpf_jit_comp.c | 43 ++++++++++- kernel/bpf/core.c | 28 ++++++- kernel/bpf/verifier.c | 150 +++++++++++++++++++++++++++++------- 3 files changed, 190 insertions(+), 31 deletions(-)
arch/x86/net/bpf_jit_comp.c +40 -3
@@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) *pprog = prog; } +static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); + }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); +EMIT3 (add_2mod (0x0f , src_reg, dst_reg),0xbf , +add_2reg (0xC0 , src_reg, dst_reg)); + } + } + + *pprog = prog; +} +/* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */ static void emit_insn_suffix (u8 **pprog,u32 ptr_reg,u32 val_reg,int off) {
@@ -1051,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ALU64 |BPF_MOV |BPF_X :case BPF_ALU |BPF_MOV |BPF_X : -emit_mov_reg (&prog, -BPF_CLASS (insn->code) ==BPF_ALU64 , - dst_reg, src_reg); +if (insn->off ==0 ) +emit_mov_reg (&prog, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg); +else +emit_movsx_reg (&prog, insn->off, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg);break ;/* neg dst */
kernel/bpf/core.c +26 -2
@@ -61,6 +61,7 @@#define AX regs[BPF_REG_AX]#define ARG1 regs[BPF_REG_ARG1]#define CTX regs[BPF_REG_CTX] +#define OFF insn->off#define IMM insn->immstruct bpf_mem_alloc bpf_global_ma;
@@ -1736,13 +1737,36 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)DST = -DST ;CONT ; ALU_MOV_X: -DST = (u32 )SRC ; +switch (OFF ) { +case 0 : +DST = (u32 )SRC ; +break ; +case 8 : +DST = (u32 )(s8 )SRC ; +break ; +case 16 : +DST = (u32 )(s16 )SRC ; +break ; + }CONT ; ALU_MOV_K:DST = (u32 )IMM ;CONT ; ALU64_MOV_X: -DST =SRC ; +switch (OFF ) { +case 0 : +DST =SRC ; +break ; +case 8 : +DST = (s8 )SRC ; +break ; +case 16 : +DST = (s16 )SRC ; +break ; +case 32 : +DST = (s32 )SRC ; +break ; + }CONT ; ALU64_MOV_K:DST =IMM ;
kernel/bpf/verifier.c +124 -26
185 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns
Quoted 36 lines · Yonghong Song
Quoted 36 lines
Add interpreter/jit support for new sign-extension mov insns. The original 'MOV' insn is extended to support signed version for both ALU and ALU64 operations. For ALU mode, the
insn->offvalue of 8 or 16 indicates sign-extension from 8- or 16-bit value to 32-bit value. For ALU64 mode, theinsn->offvalue of 8/16/32 indicates sign-extension from 8-, 16- or 32-bit value to 64-bit value.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 43 ++++++++++- kernel/bpf/core.c | 28 ++++++- kernel/bpf/verifier.c | 150 +++++++++++++++++++++++++++++------- 3 files changed, 190 insertions(+), 31 deletions(-)arch/x86/net/bpf_jit_comp.c +13
-0@@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) *pprog = prog; } +static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg));
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 724
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +8
-0+else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg));
Nit: As far as I understand 4-126 Vol. 2B of [1]
the 0x40 prefix (REX prefix) is optional here
(same as implemented below for num_bits == 16).[1] https://cdrdv2.intel.com/v1/dl/getContent/671200
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 728
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+ }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,
Nit: Basing on the same manual I don't understand why
add_2mod(0x0f, src_reg, dst_reg) is used, '0xf' should suffice (but I tried it both ways and it works...).
Quoted 261 lines
Quoted 261 lines
arch/x86/net/bpf_jit_comp.c +15
-3+ add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog = prog; +} +/* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */ static void emit_insn_suffix (u8 **pprog,u32 ptr_reg,u32 val_reg,int off) {@@ -1051,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ALU64 |BPF_MOV |BPF_X :case BPF_ALU |BPF_MOV |BPF_X : -emit_mov_reg (&prog, -BPF_CLASS (insn->code) ==BPF_ALU64 , - dst_reg, src_reg); +if (insn->off ==0 ) +emit_mov_reg (&prog, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg); +else +emit_movsx_reg (&prog, insn->off, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg);break ;/* neg dst */ kernel/bpf/core.c +26
-2@@ -61,6 +61,7 @@#define AX regs[BPF_REG_AX]#define ARG1 regs[BPF_REG_ARG1]#define CTX regs[BPF_REG_CTX] +#define OFF insn->off#define IMM insn->immstruct bpf_mem_alloc bpf_global_ma;@@ -1736,13 +1737,36 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)DST = -DST ;CONT ; ALU_MOV_X: -DST = (u32 )SRC ; +switch (OFF ) { +case 0 : +DST = (u32 )SRC ; +break ; +case 8 : +DST = (u32 )(s8 )SRC ; +break ; +case 16 : +DST = (u32 )(s16 )SRC ; +break ; + }CONT ; ALU_MOV_K:DST = (u32 )IMM ;CONT ; ALU64_MOV_X: -DST =SRC ; +switch (OFF ) { +case 0 : +DST =SRC ; +break ; +case 8 : +DST = (s8 )SRC ; +break ; +case 16 : +DST = (s16 )SRC ; +break ; +case 32 : +DST = (s32 )SRC ; +break ; + }CONT ; ALU64_MOV_K:DST =IMM ;kernel/bpf/verifier.c +115
-26@@ -3421,7 +3421,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,return 0 ;if (opcode ==BPF_MOV ) {if (BPF_SRC (insn->code) ==BPF_X ) { -/* dreg = sreg +/* dreg = sreg or dreg = (s8, s16, s32)sreg * dreg needs precision after this insn * sreg needs precision before this insn */ @@ -5866,6 +5866,64 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size)set_sext64_default_val (reg, size); } +static void set_sext32_default_val (struct bpf_reg_state *reg,int size) +{ +if (size ==1 ) { + reg->s32_min_value =S8_MIN ; + reg->s32_max_value =S8_MAX ; + }else { +/* size == 2 */ + reg->s32_min_value =S16_MIN ; + reg->s32_max_value =S16_MAX ; + } + reg->u32_min_value =0 ; + reg->u32_max_value =U32_MAX ; +} + +static void coerce_subreg_to_size_sx (struct bpf_reg_state *reg,int size) +{ +s32 init_s32_max, init_s32_min, s32_max, s32_min; +u32 top_smax_value, top_smin_value; +u32 num_bits = size *8 ; + + top_smax_value = ((u32 )reg->s32_max_value >> num_bits) << num_bits; + top_smin_value = ((u32 )reg->s32_min_value >> num_bits) << num_bits; + +if (top_smax_value != top_smin_value) + goto out; + +/* find the s32_min and s32_min after sign extension */ +if (size ==1 ) { + init_s32_max = (s8 )reg->s32_max_value; + init_s32_min = (s8 )reg->s32_min_value; + }else { +/* size == 2 */ + init_s32_max = (s16 )reg->s32_max_value; + init_s32_min = (s16 )reg->s32_min_value; + } + s32_max =max (init_s32_max, init_s32_min); + s32_min =min (init_s32_max, init_s32_min); + +if (s32_min >=0 && s32_max >=0 ) { + reg->s32_min_value = s32_min; + reg->s32_max_value = s32_max; + reg->u32_min_value =0 ; + reg->u32_max_value =U32_MAX ; +return ; + } + +if (s32_min <0 && s32_max <0 ) { + reg->s32_min_value = s32_min; + reg->s32_max_value = s32_max; + reg->u32_min_value = (u32 )s32_max; + reg->u32_max_value = (u32 )s32_min; +return ; + } + +out: +set_sext32_default_val (reg, size); +} +static bool bpf_map_is_rdonly (const struct bpf_map *map) {/* A map is considered read-only if the following condition are true: @@ -13003,11 +13061,23 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else if (opcode ==BPF_MOV ) {if (BPF_SRC (insn->code) ==BPF_X ) { -if (insn->imm !=0 || insn->off !=0 ) { +if (insn->imm !=0 ) {verbose (env,"BPF_MOV uses reserved fields\n" );return -EINVAL ; } +if (BPF_CLASS (insn->code) ==BPF_ALU ) { +if (insn->off !=0 && insn->off !=8 && insn->off !=16 ) { +verbose (env,"BPF_MOV uses reserved fields\n" ); +return -EINVAL ; + } + }else { +if (insn->off !=0 && insn->off !=8 && insn->off !=16 && insn->off !=32 ) { +verbose (env,"BPF_MOV uses reserved fields\n" ); +return -EINVAL ; + } + } +/* check src operand */ err =check_reg_arg (env, insn->src_reg,SRC_OP );if (err)@@ -13031,18 +13101,32 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) !tnum_is_const (src_reg->var_off);if (BPF_CLASS (insn->code) ==BPF_ALU64 ) { -/* case: R1 = R2 -* copy register state to dest reg -*/ -if (need_id) -/* Assign src and dst registers the same ID -* that will be used by find_equal_scalars() -* to propagate min/max range. +if (insn->off ==0 ) { +/* case: R1 = R2 +* copy register state to dest reg */ - src_reg->id = ++env->id_gen; -copy_register_state (dst_reg, src_reg); - dst_reg->live |=REG_LIVE_WRITTEN ; - dst_reg->subreg_def =DEF_NOT_SUBREG ; +if (need_id) +/* Assign src and dst registers the same ID +* that will be used by find_equal_scalars() +* to propagate min/max range. +*/ + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def =DEF_NOT_SUBREG ; + }else { +/* case: R1 = (s8, s16 s32)R2 */ +bool no_sext = src_reg->umax_value < (1ULL << (insn->off -1 )); + +if (no_sext && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +if (!no_sext) + dst_reg->id =0 ; +coerce_reg_to_size_sx (dst_reg, insn->off >>3 ); + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def =DEF_NOT_SUBREG ; + } }else {/* R1 = (u32) R2 */ if (is_pointer_value (env, insn->src_reg)) {@@ -13051,19 +13135,33 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) insn->src_reg);return -EACCES ; }else if (src_reg->type ==SCALAR_VALUE ) { -bool is_src_reg_u32 = src_reg->umax_value <=U32_MAX ; - -if (is_src_reg_u32 && need_id) - src_reg->id = ++env->id_gen; -copy_register_state (dst_reg, src_reg); -/* Make sure ID is cleared if src_reg is not in u32 range otherwise -* dst_reg min/max could be incorrectly -* propagated into src_reg by find_equal_scalars() -*/ -if (!is_src_reg_u32) - dst_reg->id =0 ; - dst_reg->live |=REG_LIVE_WRITTEN ; - dst_reg->subreg_def = env->insn_idx +1 ; +if (insn->off ==0 ) { +bool is_src_reg_u32 = src_reg->umax_value <=U32_MAX ; + +if (is_src_reg_u32 && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +/* Make sure ID is cleared if src_reg is not in u32 range otherwise +* dst_reg min/max could be incorrectly +* propagated into src_reg by find_equal_scalars() +*/ +if (!is_src_reg_u32) + dst_reg->id =0 ; + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def = env->insn_idx +1 ; + }else { +/* case: W1 = (s8, s16)W2 */ +bool no_sext = src_reg->umax_value < (1ULL << (insn->off -1 )); +
Quoted 8 lines
kernel/bpf/verifier.c static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 13163
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+if (no_sext && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +if (!no_sext) + dst_reg->id =0 ; + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def = env->insn_idx +1 ; +coerce_subreg_to_size_sx (dst_reg, insn->off >>3 );
I tried the following test program:
{
"testtesttest",
.insns = {
BPF_MOV64_IMM(BPF_REG_7, 0xffff),
{
.code = BPF_ALU | BPF_MOV | BPF_X,
.dst_reg = BPF_REG_0,
.src_reg = BPF_REG_7,
.off = 16,
.imm = 0,
},
BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 1),
BPF_EXIT_INSN(),
},
.result = ACCEPT,
.retval = 0,
},
And it produces verification log as below:
0: R1=ctx(off=0,imm=0) R10=fp0
0: (b7) r7 = 65535 ; R7_w=P65535 1: (bc) w0 = w7 ; R0_w=P65535 R7_w=P65535 2: (77) r0 >>= 1 ; R0_w=P32767
3: (95) exit
...
FAIL retval 2147483647 != 0 (run 1/1)
Note that verifier considers R0 to be 0x7FFF at 3, while actual value during execution is 0x7FFF'FFFF.
Quoted 4 lines
Quoted 4 lines
kernel/bpf/verifier.c +1
-0+ } }else {mark_reg_unknown (env, regs, insn->dst_reg);
Re: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns
Quoted 47 lines · Eduard Zingerman
Quoted 36 lines · Yonghong Song
Quoted 36 lines
Add interpreter/jit support for new sign-extension mov insns. The original 'MOV' insn is extended to support signed version for both ALU and ALU64 operations. For ALU mode, the
insn->offvalue of 8 or 16 indicates sign-extension from 8- or 16-bit value to 32-bit value. For ALU64 mode, theinsn->offvalue of 8/16/32 indicates sign-extension from 8-, 16- or 32-bit value to 64-bit value.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 43 ++++++++++- kernel/bpf/core.c | 28 ++++++- kernel/bpf/verifier.c | 150 +++++++++++++++++++++++++++++------- 3 files changed, 190 insertions(+), 31 deletions(-)arch/x86/net/bpf_jit_comp.c +13
-0@@ -701,6 +701,38 @@ static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) *pprog = prog; } +static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg));Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 724
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +8
-0+else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg));Nit: As far as I understand 4-126 Vol. 2B of [1]
Quoted 2 lines
the 0x40 prefix (REX prefix) is optional here
(same as implemented below for num_bits == 16).I think 0x40 prefix at least neededif register is from R8 - R15? I use this website to do asm/disasm experiments and did try various combinations with first 8 and later 8 registers and it seems correct results are generated.
Quoted 4 lines
Quoted 8 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 728
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+ }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,Nit: Basing on the same manual I don't understand why
add_2mod(0x0f, src_reg, dst_reg)is used, '0xf' should suffice (but I tried it both ways and it works...).
From the above online assembler website.
But I will check the doc to see whether it can be simplified.
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
arch/x86/net/bpf_jit_comp.c +15
-3+ add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog = prog; +} +/* Emit the suffix (ModR/M etc) for addressing *(ptr_reg + off) and val_reg */ static void emit_insn_suffix (u8 **pprog,u32 ptr_reg,u32 val_reg,int off) {@@ -1051,9 +1083,14 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagecase BPF_ALU64 |BPF_MOV |BPF_X :case BPF_ALU |BPF_MOV |BPF_X : -emit_mov_reg (&prog, -BPF_CLASS (insn->code) ==BPF_ALU64 , - dst_reg, src_reg); +if (insn->off ==0 ) +emit_mov_reg (&prog, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg); +else +emit_movsx_reg (&prog, insn->off, +BPF_CLASS (insn->code) ==BPF_ALU64 , + dst_reg, src_reg);break ;/* neg dst */ kernel/bpf/core.c +1
-0@@ -61,6 +61,7 @@#define AX regs[BPF_REG_AX]#define ARG1 regs[BPF_REG_ARG1]#define CTX regs[BPF_REG_CTX] +#define OFF insn->off#define IMM insn->imm
[...]
Quoted 217 lines
Quoted 182 lines
Quoted 182 lines
kernel/bpf/verifier.c +115
-26@@ -3421,7 +3421,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,return 0 ;if (opcode ==BPF_MOV ) {if (BPF_SRC (insn->code) ==BPF_X ) { -/* dreg = sreg +/* dreg = sreg or dreg = (s8, s16, s32)sreg * dreg needs precision after this insn * sreg needs precision before this insn */ @@ -5866,6 +5866,64 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size)set_sext64_default_val (reg, size); } +static void set_sext32_default_val (struct bpf_reg_state *reg,int size) +{ +if (size ==1 ) { + reg->s32_min_value =S8_MIN ; + reg->s32_max_value =S8_MAX ; + }else { +/* size == 2 */ + reg->s32_min_value =S16_MIN ; + reg->s32_max_value =S16_MAX ; + } + reg->u32_min_value =0 ; + reg->u32_max_value =U32_MAX ; +} + +static void coerce_subreg_to_size_sx (struct bpf_reg_state *reg,int size) +{ +s32 init_s32_max, init_s32_min, s32_max, s32_min; +u32 top_smax_value, top_smin_value; +u32 num_bits = size *8 ; + + top_smax_value = ((u32 )reg->s32_max_value >> num_bits) << num_bits; + top_smin_value = ((u32 )reg->s32_min_value >> num_bits) << num_bits; + +if (top_smax_value != top_smin_value) + goto out; + +/* find the s32_min and s32_min after sign extension */ +if (size ==1 ) { + init_s32_max = (s8 )reg->s32_max_value; + init_s32_min = (s8 )reg->s32_min_value; + }else { +/* size == 2 */ + init_s32_max = (s16 )reg->s32_max_value; + init_s32_min = (s16 )reg->s32_min_value; + } + s32_max =max (init_s32_max, init_s32_min); + s32_min =min (init_s32_max, init_s32_min); + +if (s32_min >=0 && s32_max >=0 ) { + reg->s32_min_value = s32_min; + reg->s32_max_value = s32_max; + reg->u32_min_value =0 ; + reg->u32_max_value =U32_MAX ; +return ; + } + +if (s32_min <0 && s32_max <0 ) { + reg->s32_min_value = s32_min; + reg->s32_max_value = s32_max; + reg->u32_min_value = (u32 )s32_max; + reg->u32_max_value = (u32 )s32_min; +return ; + } + +out: +set_sext32_default_val (reg, size); +} +static bool bpf_map_is_rdonly (const struct bpf_map *map) {/* A map is considered read-only if the following condition are true: @@ -13003,11 +13061,23 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else if (opcode ==BPF_MOV ) {if (BPF_SRC (insn->code) ==BPF_X ) { -if (insn->imm !=0 || insn->off !=0 ) { +if (insn->imm !=0 ) {verbose (env,"BPF_MOV uses reserved fields\n" );return -EINVAL ; } +if (BPF_CLASS (insn->code) ==BPF_ALU ) { +if (insn->off !=0 && insn->off !=8 && insn->off !=16 ) { +verbose (env,"BPF_MOV uses reserved fields\n" ); +return -EINVAL ; + } + }else { +if (insn->off !=0 && insn->off !=8 && insn->off !=16 && insn->off !=32 ) { +verbose (env,"BPF_MOV uses reserved fields\n" ); +return -EINVAL ; + } + } +/* check src operand */ err =check_reg_arg (env, insn->src_reg,SRC_OP );if (err)@@ -13031,18 +13101,32 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) !tnum_is_const (src_reg->var_off);if (BPF_CLASS (insn->code) ==BPF_ALU64 ) { -/* case: R1 = R2 -* copy register state to dest reg -*/ -if (need_id) -/* Assign src and dst registers the same ID -* that will be used by find_equal_scalars() -* to propagate min/max range. +if (insn->off ==0 ) { +/* case: R1 = R2 +* copy register state to dest reg */ - src_reg->id = ++env->id_gen; -copy_register_state (dst_reg, src_reg); - dst_reg->live |=REG_LIVE_WRITTEN ; - dst_reg->subreg_def =DEF_NOT_SUBREG ; +if (need_id) +/* Assign src and dst registers the same ID +* that will be used by find_equal_scalars() +* to propagate min/max range. +*/ + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def =DEF_NOT_SUBREG ; + }else { +/* case: R1 = (s8, s16 s32)R2 */ +bool no_sext = src_reg->umax_value < (1ULL << (insn->off -1 )); + +if (no_sext && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +if (!no_sext) + dst_reg->id =0 ; +coerce_reg_to_size_sx (dst_reg, insn->off >>3 ); + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def =DEF_NOT_SUBREG ; + } }else {/* R1 = (u32) R2 */ if (is_pointer_value (env, insn->src_reg)) {@@ -13051,19 +13135,33 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) insn->src_reg);return -EACCES ; }else if (src_reg->type ==SCALAR_VALUE ) { -bool is_src_reg_u32 = src_reg->umax_value <=U32_MAX ; - -if (is_src_reg_u32 && need_id) - src_reg->id = ++env->id_gen; -copy_register_state (dst_reg, src_reg); -/* Make sure ID is cleared if src_reg is not in u32 range otherwise -* dst_reg min/max could be incorrectly -* propagated into src_reg by find_equal_scalars() -*/ -if (!is_src_reg_u32) - dst_reg->id =0 ; - dst_reg->live |=REG_LIVE_WRITTEN ; - dst_reg->subreg_def = env->insn_idx +1 ; +if (insn->off ==0 ) { +bool is_src_reg_u32 = src_reg->umax_value <=U32_MAX ; + +if (is_src_reg_u32 && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +/* Make sure ID is cleared if src_reg is not in u32 range otherwise +* dst_reg min/max could be incorrectly +* propagated into src_reg by find_equal_scalars() +*/ +if (!is_src_reg_u32) + dst_reg->id =0 ; + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def = env->insn_idx +1 ; + }else { +/* case: W1 = (s8, s16)W2 */ +bool no_sext = src_reg->umax_value < (1ULL << (insn->off -1 )); +Quoted 8 lines
kernel/bpf/verifier.c static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · line 13163
Quoted 8 lines
kernel/bpf/verifier.c +8
-0+if (no_sext && need_id) + src_reg->id = ++env->id_gen; +copy_register_state (dst_reg, src_reg); +if (!no_sext) + dst_reg->id =0 ; + dst_reg->live |=REG_LIVE_WRITTEN ; + dst_reg->subreg_def = env->insn_idx +1 ; +coerce_subreg_to_size_sx (dst_reg, insn->off >>3 );I tried the following test program:
{
"testtesttest",
.insns = {
BPF_MOV64_IMM(BPF_REG_7, 0xffff),
{
.code =BPF_ALU|BPF_MOV|BPF_X,
.dst_reg =BPF_REG_0,
.src_reg =BPF_REG_7,
.off = 16,
.imm = 0,
},
BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 1),
BPF_EXIT_INSN(),
},
.result = ACCEPT,
.retval = 0,
},And it produces verification log as below:
0: R1=ctx(off=0,imm=0) R10=fp0
0: (b7) r7 = 65535 ; R7_w=P65535 1: (bc) w0 = w7 ; R0_w=P65535 R7_w=P65535 2: (77) r0 >>= 1 ; R0_w=P32767
Quoted 6 lines
3: (95) exit
...
FAIL retval 2147483647 != 0 (run 1/1)Note that verifier considers R0 to be
0x7FFFat 3, while actual value during execution is 0x7FFF'FFFF.
This is a verifier issue. Will fix.
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
kernel/bpf/verifier.c +1
-0+ } }else {mark_reg_unknown (env, regs, insn->dst_reg);
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns
On Tue, 2023-07-18 at 18:17 -0700, Yonghong Song wrote:
[...]
Quoted 26 lines
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +13
-0+static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg));Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 724
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +8
-0+else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg));Nit: As far as I understand 4-126 Vol. 2B of [1]
Quoted 2 lines
the 0x40 prefix (REX prefix) is optional here (same as implemented below for num_bits == 16).
Quoted 1 line
I think
0x40prefix at least neededif register is from R8 - R15?
Yes, please see below.
Quoted 3 lines
I use this website to do asm/disasm experiments and did try various combinations with first 8 and later 8 registers and it seems correct results are generated.
It seems all roads lead to that web-site, I used it as well :) Today I learned that the following could be used:
echo 'movsx rax,ax' | as -o /dev/null -aln -msyntax=intel -mnaked-reg
Which opens a road to scripting experiments.
Quoted 13 lines
Quoted 4 lines
Quoted 8 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 728
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+ }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,Nit: Basing on the same manual I don't understand why
add_2mod(0x0f, src_reg, dst_reg)is used, '0xf' should suffice (but I tried it both ways and it works...).
Quoted 3 lines
From the above online assembler website.
But I will check the doc to see whether it can be simplified.
I tried all combinations of r0..r9 for 64/32-bit destinations, 32/16/8 sources [1]:
- 0x40 based prefix is generated if any of the following is true:
- dst is 64 bit
- dst is ereg
- src is ereg
- dst is 32-bit and src is 'sil' (part of 'rsi', used for r2) (!) This one is surprising and web-site shows the same results. For example movsx eax,sil is encoded as 40 0F BE C6, disassembling 0F BE C6 (w/o prefix) gives movsx eax,dh.
- opcodes:
- 63 64-bit dst, 32-bit src - 0F BF 64-bit dst, 16-bit src - 0F BE 64-bit dst, 8-bit src - 0F BF 32-bit dst, 16-bit src (same as 64-bit dst) - 0F BE 32-bit dst, 8-bit src (same as 64-bit dst)
Script is at [2] (it is not particularly interesting, but in case if you want to tweak it).
[1] https://gist.github.com/eddyz87/94b35fd89f023c43dd2480e196b28ea1
[2] https://gist.github.com/eddyz87/60991379c547df11d30fa91901862227
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+ add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog = prog; +}
[...]
Re: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns
Quoted 37 lines · Eduard Zingerman
On Tue, 2023-07-18 at 18:17 -0700, Yonghong Song wrote:
[...]Quoted 26 lines
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +13
-0+static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg));Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 724
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +8
-0+else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg));Nit: As far as I understand 4-126 Vol. 2B of [1]
Quoted 2 lines
the 0x40 prefix (REX prefix) is optional here (same as implemented below for num_bits == 16).Quoted 1 line
I think
0x40prefix at least neededif register is from R8 - R15?Yes, please see below.
Quoted 3 lines
I use this website to do asm/disasm experiments and did try various combinations with first 8 and later 8 registers and it seems correct results are generated.
Quoted 6 lines
It seems all roads lead to that web-site, I used it as well :) Today I learned that the following could be used:
echo 'movsx rax,ax' | as -o
/dev/null-aln -msyntax=intel -mnaked-regWhich opens a road to scripting experiments.
This internal tool from llvm-project may also be useful:)
llvm-mc -triple=x86_64 -show-inst -x86-asm-syntax=intel -output-asm-variant=1 <<< 'movsx rax, ax'
Quoted 47 lines
Quoted 9 lines
Quoted 9 lines
[1] https://cdrdv2.intel.com/v1/dl/getContent/671200
Quoted 4 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+ }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,Quoted 7 lines
Quoted 3 lines
Nit: Basing on the same manual I don't understand why
add_2mod(0x0f, src_reg, dst_reg)is used, '0xf' should suffice (but I tried it both ways and it works...).From the above online assembler website.
But I will check the doc to see whether it can be simplified.
I tried all combinations of r0..r9 for 64/32-bit destinations, 32/16/8 sources [1]:
-0x40based prefix is generated if any of the following is true:
- dst is 64 bit
- dst is ereg
- src is ereg
- dst is 32-bit and src is 'sil' (part of 'rsi', used for r2) (!) This one is surprising and web-site shows the same results. For examplemovsx eax,silis encoded as40 0F BE C6, disassembling0F BE C6(w/o prefix) givesmovsx eax,dh.
- opcodes:- 63 64-bit dst, 32-bit src - 0F BF 64-bit dst, 16-bit src - 0F BE 64-bit dst, 8-bit src - 0F BF 32-bit dst, 16-bit src (same as 64-bit dst) - 0F BE 32-bit dst, 8-bit src (same as 64-bit dst)Script is at [2] (it is not particularly interesting, but in case if you want to tweak it).
[1] https://gist.github.com/eddyz87/94b35fd89f023c43dd2480e196b28ea1
[2] https://gist.github.com/eddyz87/60991379c547df11d30fa91901862227Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+ add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog = prog; +}[...]
Signature · 1 line
宋方睿
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 03/15] bpf: Support new sign-extension mov insns
Quoted 41 lines · Fangrui Song
Quoted 40 lines · Eduard Zingerman
On Tue, 2023-07-18 at 18:17 -0700, Yonghong Song wrote:
[...]Quoted 26 lines
Quoted 23 lines
Quoted 13 lines
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +13
-0+static void emit_movsx_reg (u8 **pprog,int num_bits,bool is64,u32 dst_reg, +u32 src_reg) +{ +u8 *prog = *pprog; + +if (is64) { +/* movs[b,w,l]q dst, src */ +if (num_bits ==8 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg)); +else if (num_bits ==16 ) +EMIT4 (add_2mod (0x48 , src_reg, dst_reg),0x0f ,0xbf , +add_2reg (0xC0 , src_reg, dst_reg));Quoted 8 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 724
Quoted 8 lines
arch/x86/net/bpf_jit_comp.c +8
-0+else if (num_bits ==32 ) +EMIT3 (add_2mod (0x48 , src_reg, dst_reg),0x63 , +add_2reg (0xC0 , src_reg, dst_reg)); + }else { +/* movs[b,w]l dst, src */ +if (num_bits ==8 ) { +EMIT4 (add_2mod (0x40 , src_reg, dst_reg),0x0f ,0xbe , +add_2reg (0xC0 , src_reg, dst_reg));Nit: As far as I understand 4-126 Vol. 2B of [1]
Quoted 2 lines
the 0x40 prefix (REX prefix) is optional here (same as implemented below for num_bits == 16).Quoted 1 line
I think
0x40prefix at least neededif register is from R8 - R15?Yes, please see below.
Quoted 3 lines
I use this website to do asm/disasm experiments and did try various combinations with first 8 and later 8 registers and it seems correct results are generated.
It seems all roads lead to that web-site, I used it as well :) Today I learned that the following could be used:
Quoted 8 lines
Quoted 3 lines
echo 'movsx rax,ax' | as -o
/dev/null-aln -msyntax=intel -mnaked-regWhich opens a road to scripting experiments.
This internal tool from llvm-project may also be useful:)
llvm-mc -triple=x86_64 -show-inst -x86-asm-syntax=intel -output-asm-variant=1 <<< 'movsx rax, ax'
Thank you, this works (with -show-encoding).
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
Quoted 8 lines
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c static void emit_mov_reg(u8 **pprog, bool is64, u32 dst_reg, u32 src_reg) · line 728
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +4
-0+ }else if (num_bits ==16 ) { +if (is_ereg (dst_reg) ||is_ereg (src_reg)) +EMIT1 (add_2mod (0x40 , src_reg, dst_reg)); + EMIT3(add_2mod(0x0f, src_reg, dst_reg), 0xbf,Nit: Basing on the same manual I don't understand why
add_2mod(0x0f, src_reg, dst_reg)is used, '0xf' should suffice (but I tried it both ways and it works...).From the above online assembler website.
But I will check the doc to see whether it can be simplified.
Quoted 10 lines
Quoted 10 lines
I tried all combinations of r0..r9 for 64/32-bit destinations, 32/16/8 sources [1]:
-0x40based prefix is generated if any of the following is true:
- dst is 64 bit
- dst is ereg
- src is ereg
- dst is 32-bit and src is 'sil' (part of 'rsi', used for r2) (!) This one is surprising and web-site shows the same results. For examplemovsx eax,silis encoded as40 0F BE C6, disassembling0F BE C6(w/o prefix) givesmovsx eax,dh.
I think I found the place in the manual that explains situation:
3.7.2.1 Register Operands in 64-Bit Mode
Register operands in 64-bit mode can be any of the following:
- ...
- 8-bit general-purpose registers: AL, BL, CL, DL, SIL, DIL, SPL, BPL, and R8B-R15B are available using REX prefixes;
AL, BL, CL, DL, AH, BH, CH, DH are available without using REX prefixes.
- ...
Vol. 1, page 3-21
https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-1-manual.pdf
Quoted 23 lines
Quoted 20 lines
- opcodes:
- 63 64-bit dst, 32-bit src - 0F BF 64-bit dst, 16-bit src - 0F BE 64-bit dst, 8-bit src - 0F BF 32-bit dst, 16-bit src (same as 64-bit dst) - 0F BE 32-bit dst, 8-bit src (same as 64-bit dst)Script is at [2] (it is not particularly interesting, but in case if you want to tweak it).
[1] https://gist.github.com/eddyz87/94b35fd89f023c43dd2480e196b28ea1
[2] https://gist.github.com/eddyz87/60991379c547df11d30fa91901862227Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/net/bpf_jit_comp.c +6
-0+ add_2reg(0xC0, src_reg, dst_reg)); + } + } + + *pprog = prog; +}[...]
PATCH v2 4/15 bpf: Support new unconditional bswap instruction
In reply to Yonghong SongThe existing 'be' and 'le' insns will do conditional bswap depends on host endianness. This patch implements unconditional bswap insns.
--- arch/x86/net/bpf_jit_comp.c | 1 + kernel/bpf/core.c | 14 ++++++++++++++ kernel/bpf/verifier.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +1 -0
@@ -1322,6 +1322,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagebreak ;case BPF_ALU |BPF_END |BPF_FROM_BE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm32) {case 16 :/* Emit 'ror %ax, 8' to swap lower 2 bytes */
kernel/bpf/core.c +14 -0
@@ -1524,6 +1524,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (ALU64 ,DIV ,X ), \INSN_3 (ALU64 ,MOD ,X ), \INSN_2 (ALU64 ,NEG ), \ +INSN_3 (ALU64 ,END ,TO_LE ), \/* Immediate based. */ \INSN_3 (ALU64 ,ADD ,K ), \ INSN_3(ALU64, SUB, K), \
@@ -1845,6 +1846,19 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)break ; }CONT ; + ALU64_END_TO_LE: +switch (IMM ) { +case 16 : +DST = (__force u16 )__swab16 (DST ); +break ; +case 32 : +DST = (__force u32 )__swab32 (DST ); +break ; +case 64 : +DST = (__force u64 )__swab64 (DST ); +break ; + } +CONT ;/* CALL */ JMP_CALL:
kernel/bpf/verifier.c +1 -1
@@ -13036,7 +13036,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {if (insn->src_reg !=BPF_REG_0 || insn->off !=0 || (insn->imm !=16 && insn->imm !=32 && insn->imm !=64 ) || -BPF_CLASS (insn->code) ==BPF_ALU64 ) { + (BPF_CLASS (insn->code) ==BPF_ALU64 &&BPF_SRC (insn->code) !=BPF_K )) {verbose (env,"BPF_END uses reserved fields\n" );return -EINVAL ; }
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 04/15] bpf: Support new unconditional bswap instruction
Quoted 5 lines · Yonghong Song
Quoted 5 lines
The existing 'be' and 'le' insns will do conditional bswap depends on host endianness. This patch implements unconditional bswap insns.
Signed-off-by Yonghong Song <[email protected]>
Note sure if this is important, but here is an observation:
function is_reg64() has the following code:
...
if (class == BPF_ALU64 || class == BPF_JMP || /* BPF_END always use BPF_ALU class. */
(class == BPF_ALU && op == BPF_END && insn->imm == 64)) return true;
...
It probably has to be updated but I'm not sure how:
- either check insn->imm == 64 for ALU64 as well;
- or just update the comment, given that instruction always sets all 64-bits.
Quoted 63 lines
Quoted 63 lines
--- arch/x86/net/bpf_jit_comp.c | 1 + kernel/bpf/core.c | 14 ++++++++++++++ kernel/bpf/verifier.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -1322,6 +1322,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagebreak ;case BPF_ALU |BPF_END |BPF_FROM_BE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm32) {case 16 :/* Emit 'ror %ax, 8' to swap lower 2 bytes */ kernel/bpf/core.c +14
-0@@ -1524,6 +1524,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (ALU64 ,DIV ,X ), \INSN_3 (ALU64 ,MOD ,X ), \INSN_2 (ALU64 ,NEG ), \ +INSN_3 (ALU64 ,END ,TO_LE ), \/* Immediate based. */ \INSN_3 (ALU64 ,ADD ,K ), \ INSN_3(ALU64, SUB, K), \@@ -1845,6 +1846,19 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)break ; }CONT ; + ALU64_END_TO_LE: +switch (IMM ) { +case 16 : +DST = (__force u16 )__swab16 (DST ); +break ; +case 32 : +DST = (__force u32 )__swab32 (DST ); +break ; +case 64 : +DST = (__force u64 )__swab64 (DST ); +break ; + } +CONT ;/* CALL */ JMP_CALL:kernel/bpf/verifier.c +1
-1@@ -13036,7 +13036,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {if (insn->src_reg !=BPF_REG_0 || insn->off !=0 || (insn->imm !=16 && insn->imm !=32 && insn->imm !=64 ) || -BPF_CLASS (insn->code) ==BPF_ALU64 ) { + (BPF_CLASS (insn->code) ==BPF_ALU64 &&BPF_SRC (insn->code) !=BPF_K )) {verbose (env,"BPF_END uses reserved fields\n" );return -EINVAL ; }
Re: [PATCH bpf-next v2 04/15] bpf: Support new unconditional bswap instruction
Quoted 17 lines · Eduard Zingerman
Quoted 5 lines · Yonghong Song
Quoted 5 lines
The existing 'be' and 'le' insns will do conditional bswap depends on host endianness. This patch implements unconditional bswap insns.
Signed-off-by Yonghong Song <[email protected]>Note sure if this is important, but here is an observation:
functionis_reg64()has the following code:... if (class == BPF_ALU64 || class == BPF_JMP || /* BPF_END always use BPF_ALU class. */ (class == BPF_ALU && op == BPF_END && insn->imm == 64)) return true; ...
Quoted 3 lines
It probably has to be updated but I'm not sure how:
- either checkinsn->imm== 64 for ALU64 as well;
- or just update the comment, given that instruction always sets all 64-bits.
I think we can remove insn->imm == 64 from condition. But I need to double check.
Quoted 65 lines
Quoted 64 lines
Quoted 64 lines
--- arch/x86/net/bpf_jit_comp.c | 1 + kernel/bpf/core.c | 14 ++++++++++++++ kernel/bpf/verifier.c | 2 +- 3 files changed, 16 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +1
-0@@ -1322,6 +1322,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_imagebreak ;case BPF_ALU |BPF_END |BPF_FROM_BE : +case BPF_ALU64 |BPF_END |BPF_FROM_LE :switch (imm32) {case 16 :/* Emit 'ror %ax, 8' to swap lower 2 bytes */ kernel/bpf/core.c +14
-0@@ -1524,6 +1524,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (ALU64 ,DIV ,X ), \INSN_3 (ALU64 ,MOD ,X ), \INSN_2 (ALU64 ,NEG ), \ +INSN_3 (ALU64 ,END ,TO_LE ), \/* Immediate based. */ \INSN_3 (ALU64 ,ADD ,K ), \ INSN_3(ALU64, SUB, K), \@@ -1845,6 +1846,19 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn)break ; }CONT ; + ALU64_END_TO_LE: +switch (IMM ) { +case 16 : +DST = (__force u16 )__swab16 (DST ); +break ; +case 32 : +DST = (__force u32 )__swab32 (DST ); +break ; +case 64 : +DST = (__force u64 )__swab64 (DST ); +break ; + } +CONT ;/* CALL */ JMP_CALL:kernel/bpf/verifier.c +1
-1@@ -13036,7 +13036,7 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {if (insn->src_reg !=BPF_REG_0 || insn->off !=0 || (insn->imm !=16 && insn->imm !=32 && insn->imm !=64 ) || -BPF_CLASS (insn->code) ==BPF_ALU64 ) { + (BPF_CLASS (insn->code) ==BPF_ALU64 &&BPF_SRC (insn->code) !=BPF_K )) {verbose (env,"BPF_END uses reserved fields\n" );return -EINVAL ; }
PATCH v2 5/15 bpf: Support new signed div/mod instructions.
In reply to Yonghong SongAdd interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus insn->off == 1. Also add basic verifier support to ensure new insns get accepted.
--- arch/x86/net/bpf_jit_comp.c | 27 +++++++---- kernel/bpf/core.c | 96 ++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 6 ++- 3 files changed, 103 insertions(+), 26 deletions(-)
arch/x86/net/bpf_jit_comp.c +19 -8
@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 ); -/* div src_reg */ -maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cltd */ +else +EMIT2 (0x48 ,0x99 );/* cqto */ + +/* idiv src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF8 , src_reg)); + }if (BPF_OP (insn->code) ==BPF_MOD && dst_reg !=BPF_REG_3 )
kernel/bpf/core.c +80 -16
116 lines not on this page · raw message
kernel/bpf/verifier.c +4 -2
@@ -13192,7 +13192,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {/* all other ALU ops: and, sub, xor, add, ... */ if (BPF_SRC (insn->code) ==BPF_X ) { -if (insn->imm !=0 || insn->off !=0 ) { +if (insn->imm !=0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }
@@ -13201,7 +13202,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)if (err)return err; }else { -if (insn->src_reg !=BPF_REG_0 || insn->off !=0 ) { +if (insn->src_reg !=BPF_REG_0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 05/15] bpf: Support new signed div/mod instructions.
Quoted 34 lines · Yonghong Song
Add interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus
insn->off== 1. Also add basic verifier support to ensure new insns get accepted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 27 +++++++---- kernel/bpf/core.c | 96 ++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 6 ++- 3 files changed, 103 insertions(+), 26 deletions(-)arch/x86/net/bpf_jit_comp.c +6
-6@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 ); -/* div src_reg */
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1211
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +8
-2-maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cltd */ +else +EMIT2 (0x48 ,0x99 );/* cqto */
Nitpick: I can't find names cltd/cqto in the Intel instruction manual,
instead it uses names cdq/cqo for these instructions. (See Vol. 2A pages 3-315 and 3-497)
Quoted 153 lines
arch/x86/net/bpf_jit_comp.c +5
-0+ +/* idiv src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF8 , src_reg)); + }if (BPF_OP (insn->code) ==BPF_MOD && dst_reg !=BPF_REG_3 )kernel/bpf/core.c +80
-16@@ -1789,36 +1789,100 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) (*(s64 *) &DST ) >>=IMM ;CONT ; ALU64_MOD_X: -div64_u64_rem (DST ,SRC , &AX ); -DST =AX ; +switch (OFF ) { +case 0 : +div64_u64_rem (DST ,SRC , &AX ); +DST =AX ; +break ; +case 1 : +AX =div64_s64 (DST ,SRC ); +DST =DST -AX *SRC ; +break ; + }CONT ; ALU_MOD_X: -AX = (u32 )DST ; -DST =do_div (AX , (u32 )SRC ); +switch (OFF ) { +case 0 : +AX = (u32 )DST ; +DST =do_div (AX , (u32 )SRC ); +break ; +case 1 : +AX = (s32 )DST ; +DST = (u32 )do_div (AX , (s32 )SRC ); +break ; + }CONT ; ALU64_MOD_K: -div64_u64_rem (DST ,IMM , &AX ); -DST =AX ; +switch (OFF ) { +case 0 : +div64_u64_rem (DST ,IMM , &AX ); +DST =AX ; +break ; +case 1 : +AX =div64_s64 (DST ,IMM ); +DST =DST -AX *IMM ; +break ; + }CONT ; ALU_MOD_K: -AX = (u32 )DST ; -DST =do_div (AX , (u32 )IMM ); +switch (OFF ) { +case 0 : +AX = (u32 )DST ; +DST =do_div (AX , (u32 )IMM ); +break ; +case 1 : +AX = (s32 )DST ; +DST = (u32 )do_div (AX , (s32 )IMM ); +break ; + }CONT ; ALU64_DIV_X: -DST =div64_u64 (DST ,SRC ); +switch (OFF ) { +case 0 : +DST =div64_u64 (DST ,SRC ); +break ; +case 1 : +DST =div64_s64 (DST ,SRC ); +break ; + }CONT ; ALU_DIV_X: -AX = (u32 )DST ; -do_div (AX , (u32 )SRC ); -DST = (u32 )AX ; +switch (OFF ) { +case 0 : +AX = (u32 )DST ; +do_div (AX , (u32 )SRC ); +DST = (u32 )AX ; +break ; +case 1 : +AX = (s32 )DST ; +do_div (AX , (s32 )SRC ); +DST = (u32 )AX ; +break ; + }CONT ; ALU64_DIV_K: -DST =div64_u64 (DST ,IMM ); +switch (OFF ) { +case 0 : +DST =div64_u64 (DST ,IMM ); +break ; +case 1 : +DST =div64_s64 (DST ,IMM ); +break ; + }CONT ; ALU_DIV_K: -AX = (u32 )DST ; -do_div (AX , (u32 )IMM ); -DST = (u32 )AX ; +switch (OFF ) { +case 0 : +AX = (u32 )DST ; +do_div (AX , (u32 )IMM ); +DST = (u32 )AX ; +break ; +case 1 : +AX = (s32 )DST ; +do_div (AX , (s32 )IMM ); +DST = (u32 )AX ; +break ; + }CONT ; ALU_END_TO_BE:switch (IMM ) {kernel/bpf/verifier.c +4
-2@@ -13192,7 +13192,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) }else {/* all other ALU ops: and, sub, xor, add, ... */ if (BPF_SRC (insn->code) ==BPF_X ) { -if (insn->imm !=0 || insn->off !=0 ) { +if (insn->imm !=0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }@@ -13201,7 +13202,8 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn)if (err)return err; }else { -if (insn->src_reg !=BPF_REG_0 || insn->off !=0 ) { +if (insn->src_reg !=BPF_REG_0 || insn->off >1 || + (insn->off ==1 && opcode !=BPF_MOD && opcode !=BPF_DIV )) {verbose (env,"BPF_ALU uses reserved fields\n" );return -EINVAL ; }
Re: [PATCH bpf-next v2 05/15] bpf: Support new signed div/mod instructions.
Quoted 34 lines · Eduard Zingerman
Quoted 33 lines · Yonghong Song
Add interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus
insn->off== 1. Also add basic verifier support to ensure new insns get accepted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 27 +++++++---- kernel/bpf/core.c | 96 ++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 6 ++- 3 files changed, 103 insertions(+), 26 deletions(-)arch/x86/net/bpf_jit_comp.c +6
-5@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 );
Quoted 15 lines
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1211
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +8
-3-/* div src_reg */ -maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cltd */ +else +EMIT2 (0x48 ,0x99 );/* cqto */ Nitpick: I can't find names cltd/cqto in the Intel instruction manual,
instead it uses names cdq/cqo for these instructions. (See Vol. 2A pages 3-315 and 3-497)
I got these asm names from
https://defuse.ca/online-x86-assembler.htm
I will check the Intel insn manual and make the change accordingly.
Quoted 9 lines
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c +5
-0+ +/* idiv src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF8 , src_reg)); + }if (BPF_OP (insn->code) ==BPF_MOD && dst_reg !=BPF_REG_3 )
[...]
Re: [PATCH bpf-next v2 05/15] bpf: Support new signed div/mod instructions.
Quoted 54 lines · Yonghong Song
Quoted 34 lines · Eduard Zingerman
Quoted 33 lines · Yonghong Song
Add interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus
insn->off== 1. Also add basic verifier support to ensure new insns get accepted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 27 +++++++---- kernel/bpf/core.c | 96 ++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 6 ++- 3 files changed, 103 insertions(+), 26 deletions(-)arch/x86/net/bpf_jit_comp.c +6
-5@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 );Quoted 15 lines
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1211
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +8
-3-/* div src_reg */ -maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cltd */ +else +EMIT2 (0x48 ,0x99 );/* cqto */ Nitpick: I can't find names cltd/cqto in the Intel instruction manual,
instead it uses names cdq/cqo for these instructions. (See Vol. 2A pages 3-315 and 3-497)
Quoted 4 lines
I got these asm names from
https://defuse.ca/online-x86-assembler.htm
I will check the Intel insn manual and make the change accordingly.
Heh. I've been using the same.
Most of the comments in the x86 JIT code are from there :)
and it actually returns 99 -> cdq, 4899 -> cqo
cltd/cqto must be aliases ?
Re: [PATCH bpf-next v2 05/15] bpf: Support new signed div/mod instructions.
Quoted 60 lines · Alexei Starovoitov
Quoted 54 lines · Yonghong Song
Quoted 34 lines · Eduard Zingerman
Quoted 33 lines · Yonghong Song
Add interpreter/jit support for new signed div/mod insns. The new signed div/mod instructions are encoded with unsigned div/mod instructions plus
insn->off== 1. Also add basic verifier support to ensure new insns get accepted.Signed-off-by Yonghong Song <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 27 +++++++---- kernel/bpf/core.c | 96 ++++++++++++++++++++++++++++++------- kernel/bpf/verifier.c | 6 ++- 3 files changed, 103 insertions(+), 26 deletions(-)arch/x86/net/bpf_jit_comp.c +6
-5@@ -1194,15 +1194,26 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image/* mov rax, dst_reg */ emit_mov_reg (&prog, is64,BPF_REG_0 , dst_reg); -/* -* xor edx, edx -* equivalent to 'xor rdx, rdx', but one byte less -*/ -EMIT2 (0x31 ,0xd2 ); +if (insn->off ==0 ) { +/* +* xor edx, edx +* equivalent to 'xor rdx, rdx', but one byte less +*/ +EMIT2 (0x31 ,0xd2 );Quoted 15 lines
arch/x86/net/bpf_jit_comp.c static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image · line 1211
Quoted 11 lines
arch/x86/net/bpf_jit_comp.c +8
-3-/* div src_reg */ -maybe_emit_1mod (&prog, src_reg, is64); -EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); +/* div src_reg */ +maybe_emit_1mod (&prog, src_reg, is64); +EMIT2 (0xF7 ,add_1reg (0xF0 , src_reg)); + }else { +if (BPF_CLASS (insn->code) ==BPF_ALU ) +EMIT1 (0x99 );/* cltd */ +else +EMIT2 (0x48 ,0x99 );/* cqto */ Nitpick: I can't find names cltd/cqto in the Intel instruction manual,
instead it uses names cdq/cqo for these instructions. (See Vol. 2A pages 3-315 and 3-497)Quoted 4 lines
I got these asm names from
https://defuse.ca/online-x86-assembler.htm
I will check the Intel insn manual and make the change accordingly.
Quoted 6 lines
Heh. I've been using the same.
Most of the comments in the x86 JIT code are from there :)and it actually returns 99 -> cdq, 4899 -> cqo
cltd/cqto must be aliases ?
Yes, cltd/cqto are aliases.
From llvm-project repo:
[~/work/llvm-project/llvm/lib/Target/X86 (ni23)]$ egrep -r cltd
egrep: warning: egrep is obsolescent; using grep -E
README.txt: cltd
X86InstrAsmAlias.td:def : MnemonicAlias<"cdq", "cltd", "att">;
X86InstrExtension.td: "{cltd|cdq}", []>, OpSize32, Sched<[WriteALU]>;
[~/work/llvm-project/llvm/lib/Target/X86 (ni23)]$ egrep -r cqto
egrep: warning: egrep is obsolescent; using grep -E X86InstrAsmAlias.td:def : MnemonicAlias<"cqo", "cqto", "att">;
X86InstrExtension.td: "{cqto|cqo}", []>, Sched<[WriteALU]>, Requires<[In64BitMode]>;
I will change to use cdq and cgo.
PATCH v2 6/15 bpf: Fix jit blinding with new sdiv/smov insns
In reply to Yonghong SongHandle new insns properly in bpf_jit_blind_insn() function.
--- include/linux/filter.h | 14 ++++++++++---- kernel/bpf/core.c | 4 ++-- 2 files changed, 12 insertions(+), 6 deletions(-)
include/linux/filter.h +10 -4
@@ -90,22 +90,28 @@ struct ctl_table_header;/* ALU ops on registers, bpf_add|sub|...: dst_reg += src_reg */ -#define BPF_ALU64_REG (OP ,DST ,SRC ) \ +#define BPF_ALU64_REG_OFF (OP ,DST ,SRC ,OFF ) \ ((struct bpf_insn) { \ .code = BPF_ALU64 | BPF_OP(OP) | BPF_X, \ .dst_reg = DST, \ .src_reg = SRC, \ - .off = 0, \ + .off = OFF, \ .imm = 0 }) -#define BPF_ALU32_REG (OP ,DST ,SRC ) \ +#define BPF_ALU64_REG (OP ,DST ,SRC ) \ + BPF_ALU64_REG_OFF(OP, DST, SRC, 0) + +#define BPF_ALU32_REG_OFF (OP ,DST ,SRC ,OFF ) \ ((struct bpf_insn) { \ .code = BPF_ALU | BPF_OP(OP) | BPF_X, \ .dst_reg = DST, \ .src_reg = SRC, \ - .off = 0, \ + .off = OFF, \ .imm = 0 }) +#define BPF_ALU32_REG (OP ,DST ,SRC ) \ + BPF_ALU32_REG_OFF(OP, DST, SRC, 0) +/* ALU ops on immediates, bpf_add|sub|...: dst_reg += imm32 */ #define BPF_ALU64_IMM (OP ,DST ,IMM ) \
kernel/bpf/core.c +2 -2
@@ -1272,7 +1272,7 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,case BPF_ALU |BPF_MOD |BPF_K : *to++ =BPF_ALU32_IMM (BPF_MOV ,BPF_REG_AX , imm_rnd ^ from->imm); *to++ =BPF_ALU32_IMM (BPF_XOR ,BPF_REG_AX , imm_rnd); - *to++ =BPF_ALU32_REG (from->code, from->dst_reg,BPF_REG_AX ); + *to++ =BPF_ALU32_REG_OFF (from->code, from->dst_reg,BPF_REG_AX , from->off);break ;case BPF_ALU64 |BPF_ADD |BPF_K :
@@ -1286,7 +1286,7 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from,case BPF_ALU64 |BPF_MOD |BPF_K : *to++ =BPF_ALU64_IMM (BPF_MOV ,BPF_REG_AX , imm_rnd ^ from->imm); *to++ =BPF_ALU64_IMM (BPF_XOR ,BPF_REG_AX , imm_rnd); - *to++ =BPF_ALU64_REG (from->code, from->dst_reg,BPF_REG_AX ); + *to++ =BPF_ALU64_REG_OFF (from->code, from->dst_reg,BPF_REG_AX , from->off);break ;case BPF_JMP |BPF_JEQ |BPF_K :
Signature · 1 line
2.34.1
PATCH v2 7/15 bpf: Support new 32bit offset jmp instruction
In reply to Yonghong SongAdd interpreter/jit/verifier support for 32bit offset jmp instruction. If a conditional jmp instruction needs more than 16bit offset, it can be simulated with a conditional jmp + a 32bit jmp insn.
--- arch/x86/net/bpf_jit_comp.c | 28 ++++++++++++++++++---------- kernel/bpf/core.c | 19 ++++++++++++++++--- kernel/bpf/verifier.c | 32 ++++++++++++++++++++++---------- 3 files changed, 56 insertions(+), 23 deletions(-)
arch/x86/net/bpf_jit_comp.c +18 -10
@@ -1809,16 +1809,24 @@ st: if (is_imm8(insn->off))break ;case BPF_JMP |BPF_JA : -if (insn->off ==-1 ) -/* -1 jmp instructions will always jump -* backwards two bytes. Explicitly handling -* this case avoids wasting too many passes -* when there are long sequences of replaced -* dead code. -*/ - jmp_offset =-2 ; -else - jmp_offset = addrs[i + insn->off] - addrs[i]; +case BPF_JMP32 |BPF_JA : +if (BPF_CLASS (insn->code) ==BPF_JMP ) { +if (insn->off ==-1 ) +/* -1 jmp instructions will always jump +* backwards two bytes. Explicitly handling +* this case avoids wasting too many passes +* when there are long sequences of replaced +* dead code. +*/ + jmp_offset =-2 ; +else + jmp_offset = addrs[i + insn->off] - addrs[i]; + }else { +if (insn->imm ==-1 ) + jmp_offset =-2 ; +else + jmp_offset = addrs[i + insn->imm] - addrs[i]; + }if (!jmp_offset) {/*
kernel/bpf/core.c +16 -3
@@ -373,7 +373,12 @@ static int bpf_adj_delta_to_off(struct bpf_insn *insn, u32 pos, s32 end_old, {const s32 off_min =S16_MIN , off_max =S16_MAX ;s32 delta = end_new - end_old; -s32 off = insn->off; +s32 off; + +if (insn->code == (BPF_JMP32 |BPF_JA )) + off = insn->imm; +else + off = insn->off;if (curr < pos && curr + off +1 >= end_old) off += delta;
@@ -381,8 +386,12 @@ static int bpf_adj_delta_to_off(struct bpf_insn *insn, u32 pos, s32 end_old, off -= delta;if (off < off_min || off > off_max)return -ERANGE ; -if (!probe_pass) - insn->off = off; +if (!probe_pass) { +if (insn->code == (BPF_JMP32 |BPF_JA )) + insn->imm = off; +else + insn->off = off; + }return 0 ; }
@@ -1593,6 +1602,7 @@ EXPORT_SYMBOL_GPL(__bpf_call_base);INSN_3 (JMP ,JSLE ,K ), \INSN_3 (JMP ,JSET ,K ), \INSN_2 (JMP ,JA ), \ +INSN_2 (JMP32 ,JA ), \/* Store instructions. */ \/* Register based. */ \ INSN_3(STX, MEM, B), \
@@ -1972,6 +1982,9 @@ static u64 ___bpf_prog_run(u64 *regs, const struct bpf_insn *insn) JMP_JA: insn += insn->off;CONT ; + JMP32_JA: + insn += insn->imm; +CONT ; JMP_EXIT:return BPF_R0 ;/* JMP */
kernel/bpf/verifier.c +22 -10
@@ -2855,7 +2855,10 @@ static int check_subprogs(struct bpf_verifier_env *env) goto next;if (BPF_OP (code) ==BPF_EXIT ||BPF_OP (code) ==BPF_CALL ) goto next; - off = i + insn[i].off +1 ; +if (code == (BPF_JMP32 |BPF_JA )) + off = i + insn[i].imm +1 ; +else + off = i + insn[i].off +1 ;if (off < subprog_start || off >= subprog_end) {verbose (env,"jump out of range from insn %d to %d\n" , i, off);return -EINVAL ;
@@ -2867,6 +2870,7 @@ static int check_subprogs(struct bpf_verifier_env *env)* or unconditional jump back */ if (code != (BPF_JMP |BPF_EXIT ) && + code != (BPF_JMP32 |BPF_JA ) && code != (BPF_JMP |BPF_JA )) {verbose (env,"last insn is not an exit or jmp\n" );return -EINVAL ;
@@ -14747,7 +14751,7 @@ static int visit_func_call_insn(int t, struct bpf_insn *insns,static int visit_insn (int t,struct bpf_verifier_env *env) {struct bpf_insn *insns = env->prog->insnsi, *insn = &insns[t]; -int ret; +int ret, off;if (bpf_pseudo_func (insn))return visit_func_call_insn (t, insns, env, true);
@@ -14795,14 +14799,19 @@ static int visit_insn(int t, struct bpf_verifier_env *env)if (BPF_SRC (insn->code) !=BPF_K )return -EINVAL ; +if (BPF_CLASS (insn->code) ==BPF_JMP ) + off = insn->off; +else + off = insn->imm; +/* unconditional jump with single edge */ - ret =push_insn (t, t + insn->off +1 ,FALLTHROUGH , env, + ret =push_insn (t, t + off +1 ,FALLTHROUGH , env, true);if (ret)return ret; -mark_prune_point (env, t + insn->off +1 ); -mark_jmp_point (env, t + insn->off +1 ); +mark_prune_point (env, t + off +1 ); +mark_jmp_point (env, t + off +1 );return ret;
@@ -16598,15 +16607,18 @@ static int do_check(struct bpf_verifier_env *env)mark_reg_scratched (env,BPF_REG_0 ); }else if (opcode ==BPF_JA ) {if (BPF_SRC (insn->code) !=BPF_K || - insn->imm !=0 || insn->src_reg !=BPF_REG_0 || insn->dst_reg !=BPF_REG_0 || - class ==BPF_JMP32 ) { + (class ==BPF_JMP && insn->imm !=0 ) || + (class ==BPF_JMP32 && insn->off !=0 )) {verbose (env,"BPF_JA uses reserved fields\n" );return -EINVAL ; } - env->insn_idx += insn->off +1 ; +if (class ==BPF_JMP ) + env->insn_idx += insn->off +1 ; +else + env->insn_idx += insn->imm +1 ;continue ; }else if (opcode ==BPF_EXIT ) {
@@ -17453,13 +17465,13 @@ static bool insn_is_cond_jump(u8 code) {u8 op; + op =BPF_OP (code);if (BPF_CLASS (code) ==BPF_JMP32 ) -return true; +return op !=BPF_JA ;if (BPF_CLASS (code) !=BPF_JMP )return false; - op =BPF_OP (code);return op !=BPF_JA && op !=BPF_EXIT && op !=BPF_CALL ; }
Signature · 1 line
2.34.1
PATCH v2 8/15 selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing
In reply to Yonghong SongSimilar to no-alu32 runner, a cpuv4 runner is created to test bpf programs compiled with -mcpu=v4.
The following are some num-of-insn statistics for each newer instructions, excluding naked asm (verifier_*) tests:
insn pattern # of instructions reg = (s8)reg 4 reg = (s16)reg 2 reg = (s32)reg 26 reg = *(s8 *)(reg + off) 11 reg = *(s16 *)(reg + off) 14 reg = *(s32 *)(reg + off) 15214 reg = bswap16 reg 133 reg = bswap32 reg 38 reg = bswap64 reg 14 reg s/= reg 0 reg s%= reg 0 gotol <offset> 58
Note that in llvm -mcpu=v4 implementation, the compiler is a little bit conservative about generating 'gotol' insn (32-bit branch offset) as it didn't precise count the number of insns (e.g., some insns are debug insns, etc.). Compared to old 'goto' insn, newer 'gotol' insn should have comparable verification states to 'goto' insn.
I did not collect verifier stats now since I have not really started to do proper range bound estimation with these instructions.
With current patch set, all selftests passed with -mcpu=v4 when running test_progs-cpuv4 binary.
--- tools/testing/selftests/bpf/.gitignore | 2 ++ tools/testing/selftests/bpf/Makefile | 18 ++++++++++++++---- 2 files changed, 16 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/.gitignore +2 -0
@@ -13,6 +13,7 @@ test_dev_cgroup /test_progs /test_progs-no_alu32 /test_progs-bpf_gcc +/test_progs-cpuv4 test_verifier_log feature test_sock
@@ -36,6 +37,7 @@ test_cpp *.lskel.h /no_alu32 /bpf_gcc +/cpuv4 /host-tools /tools /runqslower
tools/testing/selftests/bpf/Makefile +14 -4
@@ -44,7 +44,7 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_sock test_sockmap get_cgroup_id_user \ test_cgroup_storage \ test_tcpnotify_user test_sysctl \ - test_progs-no_alu32 + test_progs-no_alu32 test_progs-cpuv4 # Also test bpf-gcc, if present ifneq ($(BPF_GCC),)
@@ -383,6 +383,11 @@ define CLANG_NOALU32_BPF_BUILD_RULE $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) $(Q)$(CLANG) $3 -O2 --target=bpf -c $1 -mcpu=v2 -o $2 endef +# Similar to CLANG_BPF_BUILD_RULE, but with cpu-v4 +define CLANG_CPUV4_BPF_BUILD_RULE + $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) + $(Q)$(CLANG) $3 -O2 -target bpf -c $1 -mcpu=v4 -o $2 +endef # Build BPF object using GCC define GCC_BPF_BUILD_RULE $(call msg,GCC-BPF,$(TRUNNER_BINARY),$2)
@@ -425,7 +430,7 @@ LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(ske # $eval()) and pass control to DEFINE_TEST_RUNNER_RULES. # Parameters: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER TRUNNER_OUTPUT := $(OUTPUT)$(if $2,/)$2
@@ -453,7 +458,7 @@ endef # Using TRUNNER_XXX variables, provided by callers of DEFINE_TEST_RUNNER and # set up by DEFINE_TEST_RUNNER itself, create test runner build rules with: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER_RULES ifeq ($($(TRUNNER_OUTPUT)-dir),)
@@ -584,6 +589,11 @@ TRUNNER_BPF_BUILD_RULE := CLANG_NOALU32_BPF_BUILD_RULE TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) $(eval $(call DEFINE_TEST_RUNNER,test_progs,no_alu32)) +# Define test_progs-cpuv4 test runner. +TRUNNER_BPF_BUILD_RULE := CLANG_CPUV4_BPF_BUILD_RULE +TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) +$(eval $(call DEFINE_TEST_RUNNER,test_progs,cpuv4)) + # Define test_progs BPF-GCC-flavored test runner. ifneq ($(BPF_GCC),) TRUNNER_BPF_BUILD_RULE := GCC_BPF_BUILD_RULE
@@ -681,7 +691,7 @@ EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \ prog_tests/tests.h map_tests/tests.h verifier/tests.h \ feature bpftool \ $(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h *.subskel.h \ - no_alu32 bpf_gcc bpf_testmod.ko \ + no_alu32 cpuv4 bpf_gcc bpf_testmod.ko \ liburandom_read.so) .PHONY: docs docs-clean
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 08/15] selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing
Quoted 73 lines · Yonghong Song
Similar to no-alu32 runner, a cpuv4 runner is created to test bpf programs compiled with -mcpu=v4.
The following are some num-of-insn statistics for each newer instructions, excluding naked asm (verifier_*) tests:
insn pattern # of instructions reg = (s8)reg 4 reg = (s16)reg 2 reg = (s32)reg 26 reg = *(s8 *)(reg + off) 11 reg = *(s16 *)(reg + off) 14 reg = *(s32 *)(reg + off) 15214 reg = bswap16 reg 133 reg = bswap32 reg 38 reg = bswap64 reg 14 reg s/= reg 0 reg s%= reg 0 gotol <offset> 58Note that in llvm -mcpu=v4 implementation, the compiler is a little bit conservative about generating 'gotol' insn (32-bit branch offset) as it didn't precise count the number of insns (e.g., some insns are debug insns, etc.). Compared to old 'goto' insn, newer 'gotol' insn should have comparable verification states to 'goto' insn.
I did not collect verifier stats now since I have not really started to do proper range bound estimation with these instructions.
With current patch set, all selftests passed with -mcpu=v4 when running test_progs-cpuv4 binary.
Signed-off-by Yonghong Song <[email protected]>--- tools/testing/selftests/bpf/.gitignore | 2 ++ tools/testing/selftests/bpf/Makefile | 18 ++++++++++++++---- 2 files changed, 16 insertions(+), 4 deletions(-)tools/testing/selftests/bpf/.gitignore +2
-0@@ -13,6 +13,7 @@ test_dev_cgroup /test_progs /test_progs-no_alu32 /test_progs-bpf_gcc +/test_progs-cpuv4 test_verifier_log feature test_sock@@ -36,6 +37,7 @@ test_cpp *.lskel.h /no_alu32 /bpf_gcc +/cpuv4 /host-tools /tools /runqslowertools/testing/selftests/bpf/Makefile +1
-1@@ -44,7 +44,7 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_sock test_sockmap get_cgroup_id_user \ test_cgroup_storage \ test_tcpnotify_user test_sysctl \ - test_progs-no_alu32 + test_progs-no_alu32 test_progs-cpuv4 # Also test bpf-gcc, if present ifneq ($(BPF_GCC),)
tools/testing/selftests/bpf/Makefile define CLANG_NOALU32_BPF_BUILD_RULE · line 390
Quoted 8 lines
tools/testing/selftests/bpf/Makefile +5
-0@@ -383,6 +383,11 @@ define CLANG_NOALU32_BPF_BUILD_RULE $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) $(Q)$(CLANG) $3 -O2 --target=bpf -c $1 -mcpu=v2 -o $2 endef +# Similar to CLANG_BPF_BUILD_RULE, but with cpu-v4 +define CLANG_CPUV4_BPF_BUILD_RULE + $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) + $(Q)$(CLANG) $3 -O2 -target bpf -c $1 -mcpu=v4 -o $2 +endef
Use --target= (Clang 3.4 preferred form) for new code :)
Quoted 45 lines
tools/testing/selftests/bpf/Makefile +8
-4# Build BPF object using GCC define GCC_BPF_BUILD_RULE $(call msg,GCC-BPF,$(TRUNNER_BINARY),$2)@@ -425,7 +430,7 @@ LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(ske # $eval()) and pass control to DEFINE_TEST_RUNNER_RULES. # Parameters: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER TRUNNER_OUTPUT := $(OUTPUT)$(if $2,/)$2@@ -453,7 +458,7 @@ endef # Using TRUNNER_XXX variables, provided by callers of DEFINE_TEST_RUNNER and # set up by DEFINE_TEST_RUNNER itself, create test runner build rules with: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER_RULES ifeq ($($(TRUNNER_OUTPUT)-dir),)@@ -584,6 +589,11 @@ TRUNNER_BPF_BUILD_RULE := CLANG_NOALU32_BPF_BUILD_RULE TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) $(eval $(call DEFINE_TEST_RUNNER,test_progs,no_alu32)) +# Define test_progs-cpuv4 test runner. +TRUNNER_BPF_BUILD_RULE := CLANG_CPUV4_BPF_BUILD_RULE +TRUNNER_BPF_CFLAGS := $(BPF_CFLAGS) $(CLANG_CFLAGS) +$(eval $(call DEFINE_TEST_RUNNER,test_progs,cpuv4)) + # Define test_progs BPF-GCC-flavored test runner. ifneq ($(BPF_GCC),) TRUNNER_BPF_BUILD_RULE := GCC_BPF_BUILD_RULE@@ -681,7 +691,7 @@ EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) $(HOST_SCRATCH_DIR) \ prog_tests/tests.h map_tests/tests.h verifier/tests.h \ feature bpftool \ $(addprefix $(OUTPUT)/,*.o *.skel.h *.lskel.h *.subskel.h \ - no_alu32 bpf_gcc bpf_testmod.ko \ + no_alu32 cpuv4 bpf_gcc bpf_testmod.ko \ liburandom_read.so) .PHONY: docs docs-clean --2.34.1
Signature · 1 line
宋方睿
Re: [PATCH bpf-next v2 08/15] selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing
Quoted 71 lines · Fangrui Song
Quoted 70 lines · Yonghong Song
Similar to no-alu32 runner, a cpuv4 runner is created to test bpf programs compiled with -mcpu=v4.
The following are some num-of-insn statistics for each newer instructions, excluding naked asm (verifier_*) tests:
insn pattern # of instructions reg = (s8)reg 4 reg = (s16)reg 2 reg = (s32)reg 26 reg = *(s8 *)(reg + off) 11 reg = *(s16 *)(reg + off) 14 reg = *(s32 *)(reg + off) 15214 reg = bswap16 reg 133 reg = bswap32 reg 38 reg = bswap64 reg 14 reg s/= reg 0 reg s%= reg 0 gotol <offset> 58Note that in llvm -mcpu=v4 implementation, the compiler is a little bit conservative about generating 'gotol' insn (32-bit branch offset) as it didn't precise count the number of insns (e.g., some insns are debug insns, etc.). Compared to old 'goto' insn, newer 'gotol' insn should have comparable verification states to 'goto' insn.
I did not collect verifier stats now since I have not really started to do proper range bound estimation with these instructions.
With current patch set, all selftests passed with -mcpu=v4 when running test_progs-cpuv4 binary.
Signed-off-by Yonghong Song <[email protected]>--- tools/testing/selftests/bpf/.gitignore | 2 ++ tools/testing/selftests/bpf/Makefile | 18 ++++++++++++++---- 2 files changed, 16 insertions(+), 4 deletions(-)tools/testing/selftests/bpf/.gitignore +2
-0@@ -13,6 +13,7 @@ test_dev_cgroup /test_progs /test_progs-no_alu32 /test_progs-bpf_gcc +/test_progs-cpuv4 test_verifier_log feature test_sock@@ -36,6 +37,7 @@ test_cpp *.lskel.h /no_alu32 /bpf_gcc +/cpuv4 /host-tools /tools /runqslowertools/testing/selftests/bpf/Makefile +1
-1@@ -44,7 +44,7 @@ TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_lpm_map test test_sock test_sockmap get_cgroup_id_user \ test_cgroup_storage \ test_tcpnotify_user test_sysctl \ - test_progs-no_alu32 + test_progs-no_alu32 test_progs-cpuv4
Quoted 13 lines
tools/testing/selftests/bpf/Makefile define CLANG_NOALU32_BPF_BUILD_RULE · line 390
Quoted 11 lines
tools/testing/selftests/bpf/Makefile +5
-0# Also test bpf-gcc, if present ifneq ($(BPF_GCC),)@@ -383,6 +383,11 @@ define CLANG_NOALU32_BPF_BUILD_RULE $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) $(Q)$(CLANG) $3 -O2 --target=bpf -c $1 -mcpu=v2 -o $2 endef +# Similar to CLANG_BPF_BUILD_RULE, but with cpu-v4 +define CLANG_CPUV4_BPF_BUILD_RULE + $(call msg,CLNG-BPF,$(TRUNNER_BINARY),$2) + $(Q)$(CLANG) $3 -O2 -target bpf -c $1 -mcpu=v4 -o $2 +endefUse --target= (Clang 3.4 preferred form) for new code :)
Thanks for reminding me this. Will fix in the next revision. Using '-target bpf' for too long so forgot '--target=bpf' although just learned it a few weeks ago.
Quoted 11 lines
Quoted 11 lines
tools/testing/selftests/bpf/Makefile +1
-1# Build BPF object using GCC define GCC_BPF_BUILD_RULE $(call msg,GCC-BPF,$(TRUNNER_BINARY),$2)@@ -425,7 +430,7 @@ LINKED_BPF_SRCS := $(patsubst %.bpf.o,%.c,$(foreach skel,$(LINKED_SKELS),$($(ske # $eval()) and pass control to DEFINE_TEST_RUNNER_RULES. # Parameters: # $1 - test runner base binary name (e.g., test_progs) -# $2 - test runner extra "flavor" (e.g., no_alu32, gcc-bpf, etc) +# $2 - test runner extra "flavor" (e.g., no_alu32, cpuv4, gcc-bpf, etc) define DEFINE_TEST_RUNNER
[...]
PATCH v2 9/15 selftests/bpf: Add unit tests for new sign-extension load insns
In reply to Yonghong SongAdd unit tests for new ldsx insns. The test includes sign-extension with a single value or with a value range.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_ldsx.c | 115 ++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -31,6 +31,7 @@#include "verifier_int_ptr.skel.h" #include "verifier_jeq_infer_not_null.skel.h" #include "verifier_ld_ind.skel.h" +#include "verifier_ldsx.skel.h" #include "verifier_leak_ptr.skel.h" #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h"
@@ -133,6 +134,7 @@ void test_verifier_helper_value_access(void) { RUN(verifier_helper_value_accessvoid test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }void test_verifier_ld_ind (void ) {RUN (verifier_ld_ind); } +void test_verifier_ldsx (void ) {RUN (verifier_ldsx); }void test_verifier_leak_ptr (void ) {RUN (verifier_leak_ptr); }void test_verifier_loops1 (void ) {RUN (verifier_loops1); }void test_verifier_lwt (void ) {RUN (verifier_lwt); }
tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +115 -0
115 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 09/15] selftests/bpf: Add unit tests for new sign-extension load insns
Quoted 70 lines · Yonghong Song
Add unit tests for new ldsx insns. The test includes sign-extension with a single value or with a value range.
Signed-off-by Yonghong Song <[email protected]>--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_ldsx.c | 115 ++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.ctools/testing/selftests/bpf/prog_tests/verifier.c +2
-0@@ -31,6 +31,7 @@#include "verifier_int_ptr.skel.h" #include "verifier_jeq_infer_not_null.skel.h" #include "verifier_ld_ind.skel.h" +#include "verifier_ldsx.skel.h" #include "verifier_leak_ptr.skel.h" #include "verifier_loops1.skel.h" #include "verifier_lwt.skel.h" @@ -133,6 +134,7 @@ void test_verifier_helper_value_access(void) { RUN(verifier_helper_value_accessvoid test_verifier_int_ptr (void ) {RUN (verifier_int_ptr); }void test_verifier_jeq_infer_not_null (void ) {RUN (verifier_jeq_infer_not_null); }void test_verifier_ld_ind (void ) {RUN (verifier_ld_ind); } +void test_verifier_ldsx (void ) {RUN (verifier_ldsx); }void test_verifier_leak_ptr (void ) {RUN (verifier_leak_ptr); }void test_verifier_loops1 (void ) {RUN (verifier_loops1); }void test_verifier_lwt (void ) {RUN (verifier_lwt); }tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +34
-0@@ -0,0 +1,115 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("socket") +__description("LDSX, S8") +__success __success_unpriv __retval(-2) +__naked void ldsx_s8 (void ) +{ + asmvolatile (" \ +r1 = 0x3fe; \ +*(u64 *)(r10 - 8) = r1; \ +r0 = *(s8 *)(r10 - 8); \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("LDSX, S16") +__success __success_unpriv __retval(-2) +__naked void ldsx_s16 (void ) +{ + asmvolatile (" \ +r1 = 0x3fffe; \ +*(u64 *)(r10 - 8) = r1; \ +r0 = *(s16 *)(r10 - 8); \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description ("LDSX, S32" )
tools/testing/selftests/bpf/progs/verifier_ldsx.c line 44
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +10
-0+__success __success_unpriv __retval(-2) +__naked void ldsx_s32 (void ) +{ + asmvolatile (" \ +r1 = 0xfffffffe; \ +*(u64 *)(r10 - 8) = r1; \ +r0 = *(s32 *)(r10 - 8); \ +exit; \ +" ::: __clobber_all); +}
Note that __retval is a 32-bit value, so if I change
r0 = *(s32 *)(r10 - 8); to
r0 = *(u32 *)(r10 - 8);
^__retval would be the same, probably better to add some shift right as a last instruction or something like that.
(Also applies to a similar test in the movsx tests).
tools/testing/selftests/bpf/progs/verifier_ldsx.c line 49
Quoted 5 lines
tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +5
-0+ +SEC("socket") +__description("LDSX, S8 range checking") +__success __success_unpriv __retval(1) +__naked void ldsx_s8_range(void)
Maybe add __log_level(2) and a few
__msg("... R1_w=scalar(smin=-128,smax=127)") ?
So that verifier range logic is checked as well.
Quoted 66 lines
tools/testing/selftests/bpf/progs/verifier_ldsx.c new file +66
-0+{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +*(u64 *)(r10 - 8) = r0; \ +r1 = *(s8 *)(r10 - 8); \ +/* r1 with s8 range */ \ +if r1 s> 0x7f goto l0_%=; \ +if r1 s< -0x80 goto l0_%=; \ +r0 = 1; \ +l1_%=: \ +exit; \ +l0_%=: \ +r0 = 2; \ +goto l1_%=; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("LDSX, S16 range checking") +__success __success_unpriv __retval(1) +__naked void ldsx_s16_range(void) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +*(u64 *)(r10 - 8) = r0; \ +r1 = *(s16 *)(r10 - 8); \ +/* r1 with s16 range */ \ +if r1 s> 0x7fff goto l0_%=; \ +if r1 s< -0x8000 goto l0_%=; \ +r0 = 1; \ +l1_%=: \ +exit; \ +l0_%=: \ +r0 = 2; \ +goto l1_%=; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +SEC("socket") +__description("LDSX, S32 range checking") +__success __success_unpriv __retval(1) +__naked void ldsx_s32_range(void) +{ + asmvolatile (" \ +call %[bpf_get_prandom_u32]; \ +*(u64 *)(r10 - 8) = r0; \ +r1 = *(s32 *)(r10 - 8); \ +/* r1 with s16 range */ \ +if r1 s> 0x7fffFFFF goto l0_%=; \ +if r1 s< -0x80000000 goto l0_%=; \ +r0 = 1; \ +l1_%=: \ +exit; \ +l0_%=: \ +r0 = 2; \ +goto l1_%=; \ +" : + : __imm(bpf_get_prandom_u32) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
PATCH v2 10/15 selftests/bpf: Add unit tests for new sign-extension mov insns
In reply to Yonghong SongAdd unit tests for movsx insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_movsx.c | 177 ++++++++++++++++++ 2 files changed, 179 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_movsx.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -41,6 +41,7 @@#include "verifier_map_ret_val.skel.h" #include "verifier_masking.skel.h" #include "verifier_meta_access.skel.h" +#include "verifier_movsx.skel.h" #include "verifier_netfilter_ctx.skel.h" #include "verifier_netfilter_retcode.skel.h" #include "verifier_prevent_map_lookup.skel.h"
@@ -144,6 +145,7 @@ void test_verifier_map_ptr_mixing(void) { RUN(verifier_map_ptr_mixing); }void test_verifier_map_ret_val (void ) {RUN (verifier_map_ret_val); }void test_verifier_masking (void ) {RUN (verifier_masking); }void test_verifier_meta_access (void ) {RUN (verifier_meta_access); } +void test_verifier_movsx (void ) {RUN (verifier_movsx); }void test_verifier_netfilter_ctx (void ) {RUN (verifier_netfilter_ctx); }void test_verifier_netfilter_retcode (void ) {RUN (verifier_netfilter_retcode); }void test_verifier_prevent_map_lookup (void ) {RUN (verifier_prevent_map_lookup); }
tools/testing/selftests/bpf/progs/verifier_movsx.c new file +177 -0
177 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v2 11/15 selftests/bpf: Add unit tests for new bswap insns
In reply to Yonghong SongAdd unit tests for bswap insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_bswap.c | 45 +++++++++++++++++++ 2 files changed, 47 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_bswap.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -11,6 +11,7 @@#include "verifier_bounds_deduction_non_const.skel.h" #include "verifier_bounds_mix_sign_unsign.skel.h" #include "verifier_bpf_get_stack.skel.h" +#include "verifier_bswap.skel.h" #include "verifier_btf_ctx_access.skel.h" #include "verifier_cfg.skel.h" #include "verifier_cgroup_inv_retcode.skel.h"
@@ -115,6 +116,7 @@ void test_verifier_bounds_deduction(void) { RUN(verifier_bounds_deduction);void test_verifier_bounds_deduction_non_const (void ) {RUN (verifier_bounds_deduction_non_const); }void test_verifier_bounds_mix_sign_unsign (void ) {RUN (verifier_bounds_mix_sign_unsign); }void test_verifier_bpf_get_stack (void ) {RUN (verifier_bpf_get_stack); } +void test_verifier_bswap (void ) {RUN (verifier_bswap); }void test_verifier_btf_ctx_access (void ) {RUN (verifier_btf_ctx_access); }void test_verifier_cfg (void ) {RUN (verifier_cfg); }void test_verifier_cgroup_inv_retcode (void ) {RUN (verifier_cgroup_inv_retcode); }
tools/testing/selftests/bpf/progs/verifier_bswap.c new file +45 -0
45 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v2 12/15 selftests/bpf: Add unit tests for new sdiv/smod insns
In reply to Yonghong SongAdd unit tests for sdiv/smod insns.
--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_sdiv.c | 763 ++++++++++++++++++ 2 files changed, 765 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -54,6 +54,7 @@#include "verifier_ringbuf.skel.h" #include "verifier_runtime_jit.skel.h" #include "verifier_scalar_ids.skel.h" +#include "verifier_sdiv.skel.h" #include "verifier_search_pruning.skel.h" #include "verifier_sock.skel.h" #include "verifier_spill_fill.skel.h"
@@ -159,6 +160,7 @@ void test_verifier_regalloc(void) { RUN(verifier_regalloc); }void test_verifier_ringbuf (void ) {RUN (verifier_ringbuf); }void test_verifier_runtime_jit (void ) {RUN (verifier_runtime_jit); }void test_verifier_scalar_ids (void ) {RUN (verifier_scalar_ids); } +void test_verifier_sdiv (void ) {RUN (verifier_sdiv); }void test_verifier_search_pruning (void ) {RUN (verifier_search_pruning); }void test_verifier_sock (void ) {RUN (verifier_sock); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }
tools/testing/selftests/bpf/progs/verifier_sdiv.c new file +763 -0
763 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRe: [PATCH bpf-next v2 12/15] selftests/bpf: Add unit tests for new sdiv/smod insns
Quoted 744 lines · Yonghong Song
Add unit tests for sdiv/smod insns.
Signed-off-by Yonghong Song <[email protected]>--- .../selftests/bpf/prog_tests/verifier.c | 2 + .../selftests/bpf/progs/verifier_sdiv.c | 763 ++++++++++++++++++ 2 files changed, 765 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.ctools/testing/selftests/bpf/prog_tests/verifier.c +2
-0@@ -54,6 +54,7 @@#include "verifier_ringbuf.skel.h" #include "verifier_runtime_jit.skel.h" #include "verifier_scalar_ids.skel.h" +#include "verifier_sdiv.skel.h" #include "verifier_search_pruning.skel.h" #include "verifier_sock.skel.h" #include "verifier_spill_fill.skel.h" @@ -159,6 +160,7 @@ void test_verifier_regalloc(void) { RUN(verifier_regalloc); }void test_verifier_ringbuf (void ) {RUN (verifier_ringbuf); }void test_verifier_runtime_jit (void ) {RUN (verifier_runtime_jit); }void test_verifier_scalar_ids (void ) {RUN (verifier_scalar_ids); } +void test_verifier_sdiv (void ) {RUN (verifier_sdiv); }void test_verifier_search_pruning (void ) {RUN (verifier_search_pruning); }void test_verifier_sock (void ) {RUN (verifier_sock); }void test_verifier_spill_fill (void ) {RUN (verifier_spill_fill); }tools/testing/selftests/bpf/progs/verifier_sdiv.c new file +709
-0@@ -0,0 +1,763 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 1") +__success __success_unpriv __retval(-20) +__naked void sdiv32_non_zero_imm_1 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 2") +__success __success_unpriv __retval(-20) +__naked void sdiv32_non_zero_imm_2 (void ) +{ + asmvolatile (" \ +w0 = 41; \ +w0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 3") +__success __success_unpriv __retval(20) +__naked void sdiv32_non_zero_imm_3(void) +{ + asmvolatile (" \ +w0 = -41; \ +w0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 4") +__success __success_unpriv __retval(-21) +__naked void sdiv32_non_zero_imm_4 (void ) +{ + asmvolatile (" \ +w0 = -42; \ +w0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 5") +__success __success_unpriv __retval(-21) +__naked void sdiv32_non_zero_imm_5 (void ) +{ + asmvolatile (" \ +w0 = 42; \ +w0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 6") +__success __success_unpriv __retval(21) +__naked void sdiv32_non_zero_imm_6(void) +{ + asmvolatile (" \ +w0 = -42; \ +w0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 7") +__success __success_unpriv __retval(21) +__naked void sdiv32_non_zero_imm_7(void) +{ + asmvolatile (" \ +w0 = 42; \ +w0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero imm divisor, check 8") +__success __success_unpriv __retval(20) +__naked void sdiv32_non_zero_imm_8(void) +{ + asmvolatile (" \ +w0 = 41; \ +w0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 1") +__success __success_unpriv __retval(-20) +__naked void sdiv32_non_zero_reg_1 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w1 = 2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 2") +__success __success_unpriv __retval(-20) +__naked void sdiv32_non_zero_reg_2 (void ) +{ + asmvolatile (" \ +w0 = 41; \ +w1 = -2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 3") +__success __success_unpriv __retval(20) +__naked void sdiv32_non_zero_reg_3(void) +{ + asmvolatile (" \ +w0 = -41; \ +w1 = -2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 4") +__success __success_unpriv __retval(-21) +__naked void sdiv32_non_zero_reg_4 (void ) +{ + asmvolatile (" \ +w0 = -42; \ +w1 = 2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 5") +__success __success_unpriv __retval(-21) +__naked void sdiv32_non_zero_reg_5 (void ) +{ + asmvolatile (" \ +w0 = 42; \ +w1 = -2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 6") +__success __success_unpriv __retval(21) +__naked void sdiv32_non_zero_reg_6(void) +{ + asmvolatile (" \ +w0 = -42; \ +w1 = -2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 7") +__success __success_unpriv __retval(21) +__naked void sdiv32_non_zero_reg_7(void) +{ + asmvolatile (" \ +w0 = 42; \ +w1 = 2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, non-zero reg divisor, check 8") +__success __success_unpriv __retval(20) +__naked void sdiv32_non_zero_reg_8(void) +{ + asmvolatile (" \ +w0 = 41; \ +w1 = 2; \ +w0 s/= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 1") +__success __success_unpriv __retval(-20) +__naked void sdiv64_non_zero_imm_1 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 2") +__success __success_unpriv __retval(-20) +__naked void sdiv64_non_zero_imm_2 (void ) +{ + asmvolatile (" \ +r0 = 41; \ +r0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 3") +__success __success_unpriv __retval(20) +__naked void sdiv64_non_zero_imm_3(void) +{ + asmvolatile (" \ +r0 = -41; \ +r0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 4") +__success __success_unpriv __retval(-21) +__naked void sdiv64_non_zero_imm_4 (void ) +{ + asmvolatile (" \ +r0 = -42; \ +r0 s/= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 5") +__success __success_unpriv __retval(-21) +__naked void sdiv64_non_zero_imm_5 (void ) +{ + asmvolatile (" \ +r0 = 42; \ +r0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero imm divisor, check 6") +__success __success_unpriv __retval(21) +__naked void sdiv64_non_zero_imm_6(void) +{ + asmvolatile (" \ +r0 = -42; \ +r0 s/= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 1") +__success __success_unpriv __retval(-20) +__naked void sdiv64_non_zero_reg_1 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r1 = 2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 2") +__success __success_unpriv __retval(-20) +__naked void sdiv64_non_zero_reg_2 (void ) +{ + asmvolatile (" \ +r0 = 41; \ +r1 = -2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 3") +__success __success_unpriv __retval(20) +__naked void sdiv64_non_zero_reg_3(void) +{ + asmvolatile (" \ +r0 = -41; \ +r1 = -2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 4") +__success __success_unpriv __retval(-21) +__naked void sdiv64_non_zero_reg_4 (void ) +{ + asmvolatile (" \ +r0 = -42; \ +r1 = 2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 5") +__success __success_unpriv __retval(-21) +__naked void sdiv64_non_zero_reg_5 (void ) +{ + asmvolatile (" \ +r0 = 42; \ +r1 = -2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV64, non-zero reg divisor, check 6") +__success __success_unpriv __retval(21) +__naked void sdiv64_non_zero_reg_6(void) +{ + asmvolatile (" \ +r0 = -42; \ +r1 = -2; \ +r0 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 1") +__success __success_unpriv __retval(-1) +__naked void smod32_non_zero_imm_1 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 2") +__success __success_unpriv __retval(1) +__naked void smod32_non_zero_imm_2(void) +{ + asmvolatile (" \ +w0 = 41; \ +w0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 3") +__success __success_unpriv __retval(-1) +__naked void smod32_non_zero_imm_3 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 4") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_imm_4(void) +{ + asmvolatile (" \ +w0 = -42; \ +w0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 5") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_imm_5(void) +{ + asmvolatile (" \ +w0 = 42; \ +w0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero imm divisor, check 6") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_imm_6(void) +{ + asmvolatile (" \ +w0 = -42; \ +w0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 1") +__success __success_unpriv __retval(-1) +__naked void smod32_non_zero_reg_1 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w1 = 2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 2") +__success __success_unpriv __retval(1) +__naked void smod32_non_zero_reg_2(void) +{ + asmvolatile (" \ +w0 = 41; \ +w1 = -2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 3") +__success __success_unpriv __retval(-1) +__naked void smod32_non_zero_reg_3 (void ) +{ + asmvolatile (" \ +w0 = -41; \ +w1 = -2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 4") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_reg_4(void) +{ + asmvolatile (" \ +w0 = -42; \ +w1 = 2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 5") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_reg_5(void) +{ + asmvolatile (" \ +w0 = 42; \ +w1 = -2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, non-zero reg divisor, check 6") +__success __success_unpriv __retval(0) +__naked void smod32_non_zero_reg_6(void) +{ + asmvolatile (" \ +w0 = -42; \ +w1 = -2; \ +w0 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 1") +__success __success_unpriv __retval(-1) +__naked void smod64_non_zero_imm_1 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 2") +__success __success_unpriv __retval(1) +__naked void smod64_non_zero_imm_2(void) +{ + asmvolatile (" \ +r0 = 41; \ +r0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 3") +__success __success_unpriv __retval(-1) +__naked void smod64_non_zero_imm_3 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 4") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_imm_4(void) +{ + asmvolatile (" \ +r0 = -42; \ +r0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 5") +__success __success_unpriv __retval(-0) +__naked void smod64_non_zero_imm_5 (void ) +{ + asmvolatile (" \ +r0 = 42; \ +r0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 6") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_imm_6(void) +{ + asmvolatile (" \ +r0 = -42; \ +r0 s%%= -2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 7") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_imm_7(void) +{ + asmvolatile (" \ +r0 = 42; \ +r0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero imm divisor, check 8") +__success __success_unpriv __retval(1) +__naked void smod64_non_zero_imm_8(void) +{ + asmvolatile (" \ +r0 = 41; \ +r0 s%%= 2; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 1") +__success __success_unpriv __retval(-1) +__naked void smod64_non_zero_reg_1 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r1 = 2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 2") +__success __success_unpriv __retval(1) +__naked void smod64_non_zero_reg_2(void) +{ + asmvolatile (" \ +r0 = 41; \ +r1 = -2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 3") +__success __success_unpriv __retval(-1) +__naked void smod64_non_zero_reg_3 (void ) +{ + asmvolatile (" \ +r0 = -41; \ +r1 = -2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 4") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_reg_4(void) +{ + asmvolatile (" \ +r0 = -42; \ +r1 = 2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 5") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_reg_5(void) +{ + asmvolatile (" \ +r0 = 42; \ +r1 = -2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 6") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_reg_6(void) +{ + asmvolatile (" \ +r0 = -42; \ +r1 = -2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 7") +__success __success_unpriv __retval(0) +__naked void smod64_non_zero_reg_7(void) +{ + asmvolatile (" \ +r0 = 42; \ +r1 = 2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, non-zero reg divisor, check 8") +__success __success_unpriv __retval(1) +__naked void smod64_non_zero_reg_8(void) +{ + asmvolatile (" \ +r0 = 41; \ +r1 = 2; \ +r0 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SDIV32, zero divisor") +__success __success_unpriv __retval(42)
tools/testing/selftests/bpf/progs/verifier_sdiv.c line 719
Quoted 10 lines
tools/testing/selftests/bpf/progs/verifier_sdiv.c new file +10
-0+__naked void sdiv32_zero_divisor (void ) +{ + asmvolatile (" \ +w0 = 42; \ +w1 = 0; \ +w2 = -1; \ +w2 s/= w1; \ +exit; \ +" ::: __clobber_all); +}
Maybe add "w0 = w2" in the end to check how division by zero modifies the DST register?
Quoted 44 lines
tools/testing/selftests/bpf/progs/verifier_sdiv.c new file +44
-0+ +SEC("socket") +__description("SDIV64, zero divisor") +__success __success_unpriv __retval(42) +__naked void sdiv64_zero_divisor(void) +{ + asmvolatile (" \ +r0 = 42; \ +r1 = 0; \ +r2 = -1; \ +r2 s/= r1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD32, zero divisor") +__success __success_unpriv __retval(42) +__naked void smod32_zero_divisor(void) +{ + asmvolatile (" \ +w0 = 42; \ +w1 = 0; \ +w2 = -1; \ +w2 s%%= w1; \ +exit; \ +" ::: __clobber_all); +} + +SEC("socket") +__description("SMOD64, zero divisor") +__success __success_unpriv __retval(42) +__naked void smod64_zero_divisor(void) +{ + asmvolatile (" \ +r0 = 42; \ +r1 = 0; \ +r2 = -1; \ +r2 s%%= r1; \ +exit; \ +" ::: __clobber_all); +} + +char _license[] SEC("license") = "GPL";
PATCH v2 13/15 selftests/bpf: Add unit tests for new gotol insn
In reply to Yonghong SongAdd unit tests for gotol insn.
--- .../selftests/bpf/prog_tests/verifier.c | 2 ++ .../selftests/bpf/progs/verifier_gotol.c | 30 +++++++++++++++++++ 2 files changed, 32 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotol.c
tools/testing/selftests/bpf/prog_tests/verifier.c +2 -0
@@ -25,6 +25,7 @@#include "verifier_direct_stack_access_wraparound.skel.h" #include "verifier_div0.skel.h" #include "verifier_div_overflow.skel.h" +#include "verifier_gotol.skel.h" #include "verifier_helper_access_var_len.skel.h" #include "verifier_helper_packet_access.skel.h" #include "verifier_helper_restricted.skel.h"
@@ -131,6 +132,7 @@ void test_verifier_direct_packet_access(void) { RUN(verifier_direct_packet_accesvoid test_verifier_direct_stack_access_wraparound (void ) {RUN (verifier_direct_stack_access_wraparound); }void test_verifier_div0 (void ) {RUN (verifier_div0); }void test_verifier_div_overflow (void ) {RUN (verifier_div_overflow); } +void test_verifier_gotol (void ) {RUN (verifier_gotol); }void test_verifier_helper_access_var_len (void ) {RUN (verifier_helper_access_var_len); }void test_verifier_helper_packet_access (void ) {RUN (verifier_helper_packet_access); }void test_verifier_helper_restricted (void ) {RUN (verifier_helper_restricted); }
tools/testing/selftests/bpf/progs/verifier_gotol.c new file +30 -0
@@ -0,0 +1,30 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/bpf.h> +#include <bpf/bpf_helpers.h> +#include "bpf_misc.h" + +SEC("socket") +__description("gotol, small_imm") +__success __success_unpriv __retval(1) +__naked void gotol_small_imm(void) +{ + asmvolatile (" \ +call %[bpf_ktime_get_ns]; \ +if r0 == 0 goto l0_%=; \ +gotol l1_%=; \ +l2_%=: \ +gotol l3_%=; \ +l1_%=: \ +r0 = 1; \ +gotol l2_%=; \ +l0_%=: \ +r0 = 2; \ +l3_%=: \ +exit; \ +" : + : __imm(bpf_ktime_get_ns) + : __clobber_all); +} + +char _license[] SEC("license") = "GPL";
Signature · 1 line
2.34.1
PATCH v2 14/15 selftests/bpf: Test ldsx with more complex cases
In reply to Yonghong SongTest reading signed rdonly map value, read/write map value, probed memory and ctx field. Without proper verifier/git handling, the test will fail.
--- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 88 +++++++++++++++++++ .../selftests/bpf/progs/test_ldsx_insn.c | 75 ++++++++++++++++ 3 files changed, 171 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/test_ldsx_insn.c
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +8 -1
@@ -75,6 +75,12 @@ bpf_testmod_test_struct_arg_6(struct bpf_testmod_struct_arg_3 *a) {return bpf_testmod_test_struct_arg_result; } +noinline int +bpf_testmod_test_arg_ptr_to_struct (struct bpf_testmod_struct_arg_1 *a) { + bpf_testmod_test_struct_arg_result = a->a; +return bpf_testmod_test_struct_arg_result; +} + __bpf_kfunc voidbpf_testmod_test_mod_kfunc (int i) {
@@ -203,7 +209,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, .off = off, .len = len, }; -struct bpf_testmod_struct_arg_1 struct_arg1 = {10 }; +struct bpf_testmod_struct_arg_1 struct_arg1 = {10 }, struct_arg1_2 = {-1 };struct bpf_testmod_struct_arg_2 struct_arg2 = {2 ,3 };struct bpf_testmod_struct_arg_3 *struct_arg3;int i =1 ;
@@ -216,6 +222,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj, (void )bpf_testmod_test_struct_arg_3 (1 ,4 , struct_arg2); (void )bpf_testmod_test_struct_arg_4 (struct_arg1,1 ,2 ,3 , struct_arg2); (void )bpf_testmod_test_struct_arg_5 (); + (void )bpf_testmod_test_arg_ptr_to_struct (&struct_arg1_2); struct_arg3 =kmalloc ((sizeof (struct bpf_testmod_struct_arg_3 ) +sizeof (int )),GFP_KERNEL );
tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c new file +88 -0
88 lines not on this page · raw message
tools/testing/selftests/bpf/progs/test_ldsx_insn.c new file +75 -0
75 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v2 15/15 docs/bpf: Add documentation for new instructions
In reply to Yonghong SongAdd documentation in instruction-set.rst for new instruction encoding and their corresponding operations. Also removed the question related to 'no BPF_SDIV' in bpf_design_QA.rst since we have BPF_SDIV insn now.
--- Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 100 ++++++++++++------ 2 files changed, 66 insertions(+), 39 deletions(-)
Documentation/bpf/bpf_design_QA.rst +0 -5
@@ -140,11 +140,6 @@ A: Because if we picked one-to-one relationship to x64 it would have made it more complicated to support on arm64 and other archs. Also it needs div-by-zero runtime check. -Q: Why there is no BPF_SDIV for signed divide operation? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A: Because it would be rarely used. llvm errors in such case and -prints a suggestion to use unsigned divide instead. - Q: Why BPF has implicit prologue and epilogue? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ A: Because architectures like sparc have register windows and in general
Documentation/bpf/standardization/instruction-set.rst +66 -34
153 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructions
Quoted 18 lines · Yonghong Song
Documentation/bpf/standardization/instruction-set.rst +0
-10@@ -154,24 +154,27 @@ otherwise identical operations. The 'code' field encodes the operation as below, where 'src' and 'dst' refer to the values of the source and destination registers, respectively. -======== ===== ========================================================== -code value description -======== ===== ========================================================== -BPF_ADD 0x00 dst += src -BPF_SUB 0x10 dst -= src -BPF_MUL 0x20 dst \*= src -BPF_DIV 0x30 dst = (src != 0) ? (dst / src) : 0 -BPF_OR 0x40 dst \|= src -BPF_AND 0x50 dst &= src -BPF_LSH 0x60 dst <<= (src & mask)
Documentation/bpf/standardization/instruction-set.rst otherwise identical operations. · line 158
Quoted 10 lines
Documentation/bpf/standardization/instruction-set.rst +2
-8-BPF_RSH 0x70 dst >>= (src & mask) -BPF_NEG 0x80 dst = -src -BPF_MOD 0x90 dst = (src != 0) ? (dst % src) : dst -BPF_XOR 0xa0 dst ^= src -BPF_MOV 0xb0 dst = src -BPF_ARSH 0xc0 sign extending dst >>= (src & mask) -BPF_END 0xd0 byte swap operations (see `Byte swap instructions`_ below) -======== ===== ========================================================== +======== ===== ============ ========================================================== +code value offset value description
How about just 'offset' ?
Quoted 30 lines
Documentation/bpf/standardization/instruction-set.rst +19
-4+======== ===== ============ ========================================================== +BPF_ADD 0x00 0 dst += src +BPF_SUB 0x10 0 dst -= src +BPF_MUL 0x20 0 dst \*= src +BPF_DIV 0x30 0 dst = (src != 0) ? (dst / src) : 0 +BPF_SDIV 0x30 1 dst = (src != 0) ? (dst s/ src) : 0 +BPF_OR 0x40 0 dst \|= src +BPF_AND 0x50 0 dst &= src +BPF_LSH 0x60 0 dst <<= (src & mask) +BPF_RSH 0x70 0 dst >>= (src & mask) +BPF_NEG 0x80 0 dst = -src +BPF_MOD 0x90 0 dst = (src != 0) ? (dst % src) : dst +BPF_SMOD 0x90 1 dst = (src != 0) ? (dst s% src) : dst +BPF_XOR 0xa0 0 dst ^= src +BPF_MOV 0xb0 0 dst = src +BPF_MOVSX 0xb0 8/16/32 dst = (s8,16,s32)src +BPF_ARSH 0xc0 0 sign extending dst >>= (src & mask) +BPF_END 0xd0 0 byte swap operations (see `Byte swap instructions`_ below) +======== ===== ============ ========================================================== Underflow and overflow are allowed during arithmetic operations, meaning the 64-bit or 32-bit value will wrap. If eBPF program execution would@@ -198,11 +201,19 @@ where '(u32)' indicates that the upper 32 bits are zeroed. dst = dst ^ imm32 -Also note that the division and modulo operations are unsigned. Thus, for -``BPF_ALU``, 'imm' is first interpreted as an unsigned 32-bit value, whereas -for ``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result -interpreted as an unsigned 64-bit value. There are no instructions for
Documentation/bpf/standardization/instruction-set.rst where '(u32)' indicates that the upper 32 bits are zeroed. · line 212
Quoted 10 lines
Documentation/bpf/standardization/instruction-set.rst +9
-1-signed division or modulo. +Note that most instructions have instruction offset of 0. But three instructions +(BPF_SDIV, BPF_SMOD, BPF_MOVSX) have non-zero offset. + +The devision and modulo operations support both unsigned and signed flavors. +For unsigned operation (BPF_DIV and BPF_MOD), for ``BPF_ALU``, 'imm' is first +interpreted as an unsigned 32-bit value, whereas for ``BPF_ALU64``, 'imm' is +first sign extended to 64 bits and the result interpreted as an unsigned 64-bit +value. For signed operation (BPF_SDIV and BPF_SMOD), for both ``BPF_ALU`` and +``BPF_ALU64``, 'imm' is interpreted as a signed value.
Probably worth clarifying that in case of S[DIV|MOD] | ALU64 the imm is sign extended from 32 to 64 and interpreted as signed 64-bit.
Documentation/bpf/standardization/instruction-set.rst where '(u32)' indicates that the upper 32 bits are zeroed. · line 216
Quoted 4 lines
Documentation/bpf/standardization/instruction-set.rst +4
-0+ +Instruction BPF_MOVSX does move operation with sign extension. For ``BPF_ALU`` +mode, 8-bit and 16-bit sign extensions to 32-bit are supported. For ``BPF_ALU64``, +8-bit, 16-bit and 32-bit sign extenstions to 64-bit are supported.
How about:
Instruction BPF_MOVSX does move operation with sign extension.BPF_ALU | MOVSX sign extendes 8-bit and 16-bit into 32-bit and upper 32-bit are zeroed. BPF_ALU64 | MOVSX sign extends 8-bit, 16-bit and 32-bit into 64-bit.
Documentation/bpf/standardization/instruction-set.rst Examples: · line 257
Quoted 4 lines
Documentation/bpf/standardization/instruction-set.rst +3
-0+``BPF_ALU64 | BPF_TO_LE | BPF_END`` with imm = 16 means:: + + dst = bswap16(dst)
Worth spelling out imm 32 and 64 too ?
Documentation/bpf/standardization/instruction-set.rst instructions that transfer data between a register and memory. · line 386
Quoted 8 lines
Documentation/bpf/standardization/instruction-set.rst +7
-0+The ``BPF_MEMSX`` mode modifier is used to encode sign-extension load +instructions that transfer data between a register and memory. + +``BPF_MEMSX | <size> | BPF_LDX`` means:: + + dst = *(sign-extension size *) (src + offset) +
How about:
``BPF_MEM | <size> | BPF_LDX`` means::
dst = *(unsigned size *) (src + offset)
``BPF_MEMSX | <size> | BPF_LDX`` means::
dst = *(signed size *) (src + offset)
Re: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructions
Quoted 23 lines · Alexei Starovoitov
Quoted 22 lines · Yonghong Song
Documentation/bpf/standardization/instruction-set.rst +0
-14@@ -154,24 +154,27 @@ otherwise identical operations. The 'code' field encodes the operation as below, where 'src' and 'dst' refer to the values of the source and destination registers, respectively. -======== ===== ========================================================== -code value description -======== ===== ========================================================== -BPF_ADD 0x00 dst += src -BPF_SUB 0x10 dst -= src -BPF_MUL 0x20 dst \*= src -BPF_DIV 0x30 dst = (src != 0) ? (dst / src) : 0 -BPF_OR 0x40 dst \|= src -BPF_AND 0x50 dst &= src -BPF_LSH 0x60 dst <<= (src & mask) -BPF_RSH 0x70 dst >>= (src & mask) -BPF_NEG 0x80 dst = -src -BPF_MOD 0x90 dst = (src != 0) ? (dst % src) : dst -BPF_XOR 0xa0 dst ^= src
Quoted 8 lines
Documentation/bpf/standardization/instruction-set.rst otherwise identical operations. · line 158
Quoted 6 lines
Documentation/bpf/standardization/instruction-set.rst +2
-4-BPF_MOV 0xb0 dst = src -BPF_ARSH 0xc0 sign extending dst >>= (src & mask) -BPF_END 0xd0 byte swap operations (see `Byte swap instructions`_ below) -======== ===== ========================================================== +======== ===== ============ ========================================================== +code value offset value descriptionHow about just 'offset' ?
Ack.
Quoted 36 lines
Quoted 36 lines
Documentation/bpf/standardization/instruction-set.rst +23
-5+======== ===== ============ ========================================================== +BPF_ADD 0x00 0 dst += src +BPF_SUB 0x10 0 dst -= src +BPF_MUL 0x20 0 dst \*= src +BPF_DIV 0x30 0 dst = (src != 0) ? (dst / src) : 0 +BPF_SDIV 0x30 1 dst = (src != 0) ? (dst s/ src) : 0 +BPF_OR 0x40 0 dst \|= src +BPF_AND 0x50 0 dst &= src +BPF_LSH 0x60 0 dst <<= (src & mask) +BPF_RSH 0x70 0 dst >>= (src & mask) +BPF_NEG 0x80 0 dst = -src +BPF_MOD 0x90 0 dst = (src != 0) ? (dst % src) : dst +BPF_SMOD 0x90 1 dst = (src != 0) ? (dst s% src) : dst +BPF_XOR 0xa0 0 dst ^= src +BPF_MOV 0xb0 0 dst = src +BPF_MOVSX 0xb0 8/16/32 dst = (s8,16,s32)src +BPF_ARSH 0xc0 0 sign extending dst >>= (src & mask) +BPF_END 0xd0 0 byte swap operations (see `Byte swap instructions`_ below) +======== ===== ============ ========================================================== Underflow and overflow are allowed during arithmetic operations, meaning the 64-bit or 32-bit value will wrap. If eBPF program execution would@@ -198,11 +201,19 @@ where '(u32)' indicates that the upper 32 bits are zeroed. dst = dst ^ imm32 -Also note that the division and modulo operations are unsigned. Thus, for -``BPF_ALU``, 'imm' is first interpreted as an unsigned 32-bit value, whereas -for ``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result -interpreted as an unsigned 64-bit value. There are no instructions for -signed division or modulo. +Note that most instructions have instruction offset of 0. But three instructions +(BPF_SDIV, BPF_SMOD, BPF_MOVSX) have non-zero offset. + +The devision and modulo operations support both unsigned and signed flavors.
Quoted 8 lines
Documentation/bpf/standardization/instruction-set.rst where '(u32)' indicates that the upper 32 bits are zeroed. · line 212
Quoted 5 lines
Documentation/bpf/standardization/instruction-set.rst +5
-0+For unsigned operation (BPF_DIV and BPF_MOD), for ``BPF_ALU``, 'imm' is first +interpreted as an unsigned 32-bit value, whereas for ``BPF_ALU64``, 'imm' is +first sign extended to 64 bits and the result interpreted as an unsigned 64-bit +value. For signed operation (BPF_SDIV and BPF_SMOD), for both ``BPF_ALU`` and +``BPF_ALU64``, 'imm' is interpreted as a signed value.Probably worth clarifying that in case of S[DIV|MOD] | ALU64 the imm is sign extended from 32 to 64 and interpreted as signed 64-bit.
I thought this before but think "'imm' is interpreted as a signed value"
might be good enough since in this case BPF_ALU and BPF_ALU64 should have the same signed 'imm' value. But what you suggested will make it more clearer. Will do.
Quoted 11 lines
Documentation/bpf/standardization/instruction-set.rst where '(u32)' indicates that the upper 32 bits are zeroed. · line 216
Quoted 5 lines
Documentation/bpf/standardization/instruction-set.rst +4
-0+ +Instruction BPF_MOVSX does move operation with sign extension. For ``BPF_ALU`` +mode, 8-bit and 16-bit sign extensions to 32-bit are supported. For ``BPF_ALU64``, +8-bit, 16-bit and 32-bit sign extenstions to 64-bit are supported.How about:
Instruction
BPF_MOVSXdoes move operation with sign extension.BPF_ALU| MOVSX sign extendes 8-bit and 16-bit into 32-bit and upper 32-bit are zeroed.BPF_ALU64| MOVSX sign extends 8-bit, 16-bit and 32-bit into 64-bit.
In previous doc, we already says that BPF_ALU operation has upper 32-bit zeroed. But since this sign extension is special, agree it is worthwhile to mention upper 32-bit zeroed explictly.
Quoted 7 lines
Documentation/bpf/standardization/instruction-set.rst Examples: · line 257
Quoted 5 lines
Documentation/bpf/standardization/instruction-set.rst +3
-0+``BPF_ALU64 | BPF_TO_LE | BPF_END`` with imm = 16 means:: + + dst = bswap16(dst)Worth spelling out imm 32 and 64 too ?
Will do.
Quoted 12 lines
Documentation/bpf/standardization/instruction-set.rst instructions that transfer data between a register and memory. · line 386
Quoted 9 lines
Documentation/bpf/standardization/instruction-set.rst +7
-0+The ``BPF_MEMSX`` mode modifier is used to encode sign-extension load +instructions that transfer data between a register and memory. + +``BPF_MEMSX | <size> | BPF_LDX`` means:: + + dst = *(sign-extension size *) (src + offset) +How about:
Quoted 7 lines
``BPF_MEM | <size> | BPF_LDX`` means::
dst = *(unsigned size *) (src + offset)
``BPF_MEMSX | <size> | BPF_LDX`` means::
dst = *(signed size *) (src + offset)
Will do.
Quoted 1 line
RE: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructions
In reply to Yonghong SongQuoted 3 lines · Yonghong Song
Add documentation in
instruction-set.rstfor new instruction encoding and their corresponding operations. Also removed the question related to 'noBPF_SDIV' inbpf_design_QA.rstsince we haveBPF_SDIVinsn now.
Why did you choose to differentiate the instruction by offset instead of using a separate opcode value? I don't think there's any other instructions that do so, and there's spare opcode values as far as I can see.
Using a separate offset works but would end up requiring another column in the IANA registry assuming we have one. So why the extra complexity and inconsistency introduced now?
Dave
Re: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructions
Quoted 6 lines · Dave Thaler
Quoted 3 lines · Yonghong Song
Add documentation in
instruction-set.rstfor new instruction encoding and their corresponding operations. Also removed the question related to 'noBPF_SDIV' inbpf_design_QA.rstsince we haveBPF_SDIVinsn now.
Quoted 7 lines
Why did you choose to differentiate the instruction by offset instead of using a separate opcode value? I don't think there's any other instructions that do so, and there's spare opcode values as far as I can see.
Using a separate offset works but would end up requiring another column in the IANA registry assuming we have one. So why the extra complexity and inconsistency introduced now?
"another column in IANA" is the last thing to worry about.
RE: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructions
In reply to Yonghong SongQuoted 233 lines
-----Original Message-----
From: Yonghong Song <[email protected]>
Sent: Wednesday, July 12, 2023 11:09 PM
To: [email protected]
Cc: Alexei Starovoitov <[email protected]>; Andrii Nakryiko <[email protected]>; Daniel Borkmann <[email protected]>; Fangrui Song <[email protected]>; [email protected]
Subject: [PATCH bpf-next v2 15/15] docs/bpf: Add documentation for new instructionsAdd documentation in
instruction-set.rstfor new instruction encoding and their corresponding operations. Also removed the question related to 'noBPF_SDIV' inbpf_design_QA.rstsince we haveBPF_SDIVinsn now.Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 100 ++++++++++++------ 2 files changed, 66 insertions(+), 39 deletions(-)Documentation/bpf/bpf_design_QA.rst +0
-0@@ -140,11 +140,6 @@ A: Because if we picked one-to-one relationship tox64 it would have made it more complicated to support on arm64 and other archs. Also it needs div-by-zero runtime check.
Documentation/bpf/bpf_design_QA.rst +0
-3-Q: Why there is no BPF_SDIV for signed divide operation? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A: Because it would be rarely used. llvm errors in such case and -prints asuggestion to use unsigned divide instead.
-
Q: Why BPF has implicit prologue and epilogue?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A: Because architectures like sparc have register windows and in general diff
--gita/Documentation/bpf/standardization/instruction-set.rstb/Documentation/bpf/standardization/instruction-set.rst
index751e657973f0..367f426d09a1100644Documentation/bpf/standardization/instruction-set.rst +0
-0@@ -154,24 +154,27 @@ otherwise identical operations. The 'code' field encodes the operation as below, where 'src' and 'dst' refer tothe values of the source and destination registers, respectively.
-======== =====
==========================================================
-code value description
-======== =====
==========================================================Documentation/bpf/standardization/instruction-set.rst +0
-14-BPF_ADD 0x00 dst += src -BPF_SUB 0x10 dst -= src -BPF_MUL 0x20 dst \*= src -BPF_DIV 0x30 dst = (src != 0) ? (dst / src) : 0 -BPF_OR 0x40 dst \|= src -BPF_AND 0x50 dst &= src -BPF_LSH 0x60 dst <<= (src & mask) -BPF_RSH 0x70 dst >>= (src & mask) -BPF_NEG 0x80 dst = -src -BPF_MOD 0x90 dst = (src != 0) ? (dst % src) : dst -BPF_XOR 0xa0 dst ^= src -BPF_MOV 0xb0 dst = src -BPF_ARSH 0xc0 sign extending dst >>= (src & mask) -BPF_END 0xd0 byte swap operations (see `Byte swap instructions`_below)
-======== =====
==========================================================
+======== ===== ============
==========================================================+code value offset value description +======== ===== ======================================================================
Documentation/bpf/standardization/instruction-set.rst +17
-0+BPF_ADD 0x00 0 dst += src +BPF_SUB 0x10 0 dst -= src +BPF_MUL 0x20 0 dst \*= src +BPF_DIV 0x30 0 dst = (src != 0) ? (dst / src) : 0 +BPF_SDIV 0x30 1 dst = (src != 0) ? (dst s/ src) : 0 +BPF_OR 0x40 0 dst \|= src +BPF_AND 0x50 0 dst &= src +BPF_LSH 0x60 0 dst <<= (src & mask) +BPF_RSH 0x70 0 dst >>= (src & mask) +BPF_NEG 0x80 0 dst = -src +BPF_MOD 0x90 0 dst = (src != 0) ? (dst % src) : dst +BPF_SMOD 0x90 1 dst = (src != 0) ? (dst s% src) : dst +BPF_XOR 0xa0 0 dst ^= src +BPF_MOV 0xb0 0 dst = src +BPF_MOVSX 0xb0 8/16/32 dst = (s8,16,s32)src +BPF_ARSH 0xc0 0 sign extending dst >>= (src & mask) +BPF_END 0xd0 0 byte swap operations (see `Byte swapinstructions`_ below)
+======== ===== ============ +==========================================================Underflow and overflow are allowed during arithmetic operations, meaning
the 64-bit or 32-bit value will wrap. If eBPF program execution would @@ - 198,11 +201,19 @@ where '(u32)' indicates that the upper 32 bits are zeroed.dst = dst ^ imm32
-Also note that the division and modulo operations are unsigned. Thus, for - ``BPF_ALU``, 'imm' is first interpreted as an unsigned 32-bit value, whereas - for ``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result - interpreted as an unsigned 64-bit value. There are no instructions for -signed division or modulo.
Documentation/bpf/standardization/instruction-set.rst +4
-0+Note that most instructions have instruction offset of 0. But three +instructions (BPF_SDIV, BPF_SMOD, BPF_MOVSX) have non-zero offset. + +The devision and modulo operations support both unsigned and signedflavors.
Documentation/bpf/standardization/instruction-set.rst +5
-0+For unsigned operation (BPF_DIV and BPF_MOD), for ``BPF_ALU``, 'imm' is +first interpreted as an unsigned 32-bit value, whereas for +``BPF_ALU64``, 'imm' is first sign extended to 64 bits and the result +interpreted as an unsigned 64-bit value. For signed operation +(BPF_SDIV and BPF_SMOD), for both ``BPF_ALU`` and ``BPF_ALU64``, 'imm'is interpreted as a signed value.
Documentation/bpf/standardization/instruction-set.rst +4
-0+ +Instruction BPF_MOVSX does move operation with sign extension. For +``BPF_ALU`` mode, 8-bit and 16-bit sign extensions to 32-bit are +supported. For ``BPF_ALU64``, 8-bit, 16-bit and 32-bit sign extenstions to64-bit are supported.
Shift operations use a mask of
0x3F(63) for 64-bit operations and0x1F(31)
for 32-bit operations.Documentation/bpf/standardization/instruction-set.rst +2
-2@@ -210,21 +221,23 @@ for 32-bit operations. Byte swap instructions ~~~~~~~~~~~~~~~~~~~~~~ -The byte swap instructions use an instruction class of ``BPF_ALU`` and a 4-bit -'code' field of ``BPF_END``. +The byte swap instructions use instruction classes of ``BPF_ALU`` and +``BPF_ALU64`` and a 4-bit 'code' field of ``BPF_END``. The byte swap instructions operate on the destination register only and donot use a separate source register or immediate value.
-The 1-bit source operand field in the opcode is used to select what byte - order the operation convert from or to:
Documentation/bpf/standardization/instruction-set.rst +3
-1+For ``BPF_ALU``, the 1-bit source operand field in the opcode is used +to select what byte order the operation convert from or to. For +``BPF_ALU64``, the 1-bit source operand field in the opcode is not used. -========= ======================================================
-source value description
-========= =====
=================================================Documentation/bpf/standardization/instruction-set.rst +0
-3-BPF_TO_LE 0x00 convert between host byte order and little endian -BPF_TO_BE 0x08 convert between host byte order and big endian -========= ======================================================
+========= ========= =====
=================================================+class source value description +========= ========= ======================================================
+BPF_ALUBPF_TO_LE0x00convert between host byte order and little
endian
+BPF_ALUBPF_TO_BE0x08convert between host byte order and big
endian+BPF_ALU64 BPF_TO_LE 0x00 do byte swap unconditionally +========= ========= =====+=================================================
The 'imm' field encodes the width of the swap operations. The following
widths are supported: 16, 32 and 64.Documentation/bpf/standardization/instruction-set.rst +4
-0@@ -239,6 +252,10 @@ Examples: dst = htobe64(dst) +``BPF_ALU64 | BPF_TO_LE | BPF_END`` with imm = 16 means:: + + dst = bswap16(dst) + Jump instructions -----------------@@ -249,7 +266,8 @@ The 'code' field encodes the operation as below: ======== ===== === ====================================================================================
code value src description notes ======== ===== === ====================================================================================
Documentation/bpf/standardization/instruction-set.rst +7
-1-BPF_JA 0x0 0x0 PC += offset BPF_JMP only +BPF_JA 0x0 0x0 PC += offset BPF_JMP class +BPF_JA 0x0 0x0 PC += imm BPF_JMP32 class BPF_JEQ 0x1 any PC += offset if dst == src BPF_JGT 0x2 any PC += offset if dst > src unsigned BPF_JGE 0x3 any PC += offset if dst >= src unsigned@@ -278,6 +296,10 @@ Example: where 's>=' indicates a signed '>=' comparison. +Note there are two flavors of BPF_JA instrions. BPF_JMP class permits +16-bit jump offset while +BPF_JMP32 permits 32-bit jump offset. A >16bit conditional jmp can be +converted to a <16bit conditional jmp plus a 32-bit unconditional jump. + Helper functions ~~~~~~~~~~~~~~~~@@ -320,6 +342,7 @@ The mode modifier is one of: BPF_ABS 0x20 legacy BPF packet access (absolute) `Legacy BPFPacket access instructions`_
BPF_IND0x40legacy BPF packet access (indirect) `Legacy BPF Packet
access instructions`_
BPF_MEM0x60regular load and store operations `Regular load and
store operations`_
+BPF_MEMSX0x80sign-extension load operations `Sign-extension
load operations`_BPF_ATOMIC 0xc0 atomic operations `Atomic operations`_ ============= ===== =================================================
Documentation/bpf/standardization/instruction-set.rst +0
-0@@ -354,6 +377,15 @@ instructions that transfer data between a registerand memory.
Where size is one of: ``BPF_B``, ``BPF_H``, ``BPF_W``, or ``BPF_DW``.
Documentation/bpf/standardization/instruction-set.rst +9
-1+The ``BPF_MEMSX`` mode modifier is used to encode sign-extension load +instructions that transfer data between a register and memory. + +``BPF_MEMSX | <size> | BPF_LDX`` means:: + + dst = *(sign-extension size *) (src + offset) + +Where size is one of: ``BPF_B``, ``BPF_H`` or ``BPF_W``. + Atomic operations ----------------- --
Quoted 2 lines
2.34.1
Adding [email protected] to the To line since this is proposing a chance to the ISA draft.
Dave
Eduard Zingerman
maintainerQuoted 11 lines · Yonghong Song
Quoted 11 lines
In previous discussion ([1]), it is agreed that we should introduce cpu version 4 (llvm flag -mcpu=v4) which contains some instructions which can simplify code, make code easier to understand, fix the existing problem, or simply for feature completeness. More specifically, the following new insns are proposed:
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offset
Quoted 6 lines
Quoted 6 lines
This patch set added kernel support for insns proposed in [1] except
BPF_STwhich already has full kernel support. Beside the above proposed insns, LLVM will generateBPF_STinsn as well under -mcpu=v4 ([2]).The patchset implements interpreter and jit support for these new insns as well as necessary verifier support.
Hi Yonghong,
Sorry for delayed response, I'm still in the middle of the series. I've tested this patch-set using updated LLVM version and have a few notes:
- kernel/bpf/disasm.c needs an update to handle new instructions;
- test_verifier test 'invalid 64-bit BPF_END' from basic_instr.c is failing because '.code = BPF_ALU64 | BPF_END | BPF_TO_LE' is now valid;
- I've looked through the usage of BPF_LDX and found that there is a function seccomp.c:seccomp_check_filter(), that directly checks possible CLASS / CODE combinations. Should this function be updated to handle new instructions?
Thanks,
Eduard
Quoted 61 lines
Quoted 61 lines
To test this patch set, you need to have latest llvm from 'main' branch of llvm-project repo and apply [2] on top of it.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://reviews.llvm.org/D144829Changelogs:
RFCv1 -> v2:
. add more verifier supports for signed extend load and mov insns. . rename some insn names to be more consistent with intel practice. . add cpuv4 test runner for test progs.
. add more unit and C tests.
. add documentation.Yonghong Song (15):
bpf: Support new sign-extension load insns
bpf: Fix sign-extension ctx member accesses
bpf: Support new sign-extension mov insns
bpf: Support new unconditional bswap instruction
bpf: Support new signed div/mod instructions.
bpf: Fix jit blinding with new sdiv/smov insns
bpf: Support new 32bit offset jmp instruction selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing selftests/bpf: Add unit tests for new sign-extension load insns selftests/bpf: Add unit tests for new sign-extension mov insns selftests/bpf: Add unit tests for new bswap insns selftests/bpf: Add unit tests for new sdiv/smod insns selftests/bpf: Add unit tests for new gotol insn selftests/bpf: Test ldsx with more complex cases docs/bpf: Add documentation for new instructionsDocumentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 100 ++- arch/x86/net/bpf_jit_comp.c | 131 ++- include/linux/filter.h | 14 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/cgroup.c | 14 +- kernel/bpf/core.c | 174 +++- kernel/bpf/verifier.c | 315 ++++++-- tools/include/uapi/linux/bpf.h | 1 + tools/testing/selftests/bpf/.gitignore | 2 + tools/testing/selftests/bpf/Makefile | 18 +- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 88 ++ .../selftests/bpf/prog_tests/verifier.c | 10 + .../selftests/bpf/progs/test_ldsx_insn.c | 75 ++ .../selftests/bpf/progs/verifier_bswap.c | 45 ++ .../selftests/bpf/progs/verifier_gotol.c | 30 + .../selftests/bpf/progs/verifier_ldsx.c | 115 +++ .../selftests/bpf/progs/verifier_movsx.c | 177 ++++ .../selftests/bpf/progs/verifier_sdiv.c | 763 ++++++++++++++++++ 20 files changed, 1929 insertions(+), 158 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/test_ldsx_insn.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_bswap.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_gotol.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_ldsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_movsx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_sdiv.c
Quoted 4 lines · Eduard Zingerman
- I've looked through the usage of
BPF_LDXand found that there is a function seccomp.c:seccomp_check_filter(), that directly checks possible CLASS / CODE combinations. Should this function be updated to handle new instructions?
This is classic bpf. Why would it change? What is the concern?
Eduard Zingerman
maintainerQuoted 7 lines · Alexei Starovoitov
Quoted 4 lines · Eduard Zingerman
- I've looked through the usage of
BPF_LDXand found that there is a function seccomp.c:seccomp_check_filter(), that directly checks possible CLASS / CODE combinations. Should this function be updated to handle new instructions?This is classic bpf. Why would it change? What is the concern?
Sorry, I missed the call to bpf_check_classic() in filter.c:bpf_prepare_filter().
Quoted 19 lines · Eduard Zingerman
Quoted 11 lines · Yonghong Song
Quoted 11 lines
In previous discussion ([1]), it is agreed that we should introduce cpu version 4 (llvm flag -mcpu=v4) which contains some instructions which can simplify code, make code easier to understand, fix the existing problem, or simply for feature completeness. More specifically, the following new insns are proposed:
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offsetQuoted 6 lines
Quoted 6 lines
This patch set added kernel support for insns proposed in [1] except
BPF_STwhich already has full kernel support. Beside the above proposed insns, LLVM will generateBPF_STinsn as well under -mcpu=v4 ([2]).The patchset implements interpreter and jit support for these new insns as well as necessary verifier support.
Quoted 6 lines
Hi Yonghong,
Sorry for delayed response, I'm still in the middle of the series. I've tested this patch-set using updated LLVM version and have a few notes:
-kernel/bpf/disasm.cneeds an update to handle new instructions;
Thanks for pointing this out. My bad. I have disasm.c patch in RFCv1, but somehow during cherry-pick process, I missed it in v2. Will add it back to next revision.
Quoted 3 lines
-
test_verifiertest 'invalid 64-bitBPF_END' frombasic_instr.cis failing because '.code =BPF_ALU64|BPF_END|BPF_TO_LE' is now valid;
I didn't run 'test_verifier'. Thanks for pointing it out. Will run it and fix the problem in the next revision.
Quoted 70 lines
- I've looked through the usage of
BPF_LDXand found that there is a function seccomp.c:seccomp_check_filter(), that directly checks possible CLASS / CODE combinations. Should this function be updated to handle new instructions?Thanks,
EduardQuoted 61 lines
Quoted 61 lines
To test this patch set, you need to have latest llvm from 'main' branch of llvm-project repo and apply [2] on top of it.
[1] https://lore.kernel.org/bpf/[email protected]/ [2] https://reviews.llvm.org/D144829Changelogs:
RFCv1 -> v2:
. add more verifier supports for signed extend load and mov insns. . rename some insn names to be more consistent with intel practice. . add cpuv4 test runner for test progs.
. add more unit and C tests.
. add documentation.Yonghong Song (15):
bpf: Support new sign-extension load insns
bpf: Fix sign-extension ctx member accesses
bpf: Support new sign-extension mov insns
bpf: Support new unconditional bswap instruction
bpf: Support new signed div/mod instructions.
bpf: Fix jit blinding with new sdiv/smov insns
bpf: Support new 32bit offset jmp instruction selftests/bpf: Add a cpuv4 test runner for cpu=v4 testing selftests/bpf: Add unit tests for new sign-extension load insns selftests/bpf: Add unit tests for new sign-extension mov insns selftests/bpf: Add unit tests for new bswap insns selftests/bpf: Add unit tests for new sdiv/smod insns selftests/bpf: Add unit tests for new gotol insn selftests/bpf: Test ldsx with more complex cases docs/bpf: Add documentation for new instructionsDocumentation/bpf/bpf_design_QA.rst | 5 - .../bpf/standardization/instruction-set.rst | 100 ++- arch/x86/net/bpf_jit_comp.c | 131 ++- include/linux/filter.h | 14 +- include/uapi/linux/bpf.h | 1 + kernel/bpf/cgroup.c | 14 +- kernel/bpf/core.c | 174 +++- kernel/bpf/verifier.c | 315 ++++++-- tools/include/uapi/linux/bpf.h | 1 + tools/testing/selftests/bpf/.gitignore | 2 + tools/testing/selftests/bpf/Makefile | 18 +- .../selftests/bpf/bpf_testmod/bpf_testmod.c | 9 +- .../selftests/bpf/prog_tests/test_ldsx_insn.c | 88 ++ .../selftests/bpf/prog_tests/verifier.c | 10 + .../selftests/bpf/progs/test_ldsx_insn.c | 75 ++ .../selftests/bpf/progs/verifier_bswap.c | 45 ++ .../selftests/bpf/progs/verifier_gotol.c | 30 + .../selftests/bpf/progs/verifier_ldsx.c | 115 +++ .../selftests/bpf/progs/verifier_movsx.c | 177 ++++ .../selftests/bpf/progs/verifier_sdiv.c | 763 ++++++++++++++++++ 20 files changed, 1929 insertions(+), 158 deletions(-)create mode 100644
tools/testing/selftests/bpf/prog_tests/test_ldsx_insn.ccreate mode 100644tools/testing/selftests/bpf/progs/test_ldsx_insn.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_bswap.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_gotol.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_ldsx.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_movsx.ccreate mode 100644tools/testing/selftests/bpf/progs/verifier_sdiv.c
Hi Yonghong.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offset
I am adding the V4 BPF instructions to binutils. Where is the precise "pseudo-c" syntax used by the new instructions documented?
For ALU sdiv/smod we are using:
rd s/= rs
rd s%= rs
For ALU32 sdiv/smod we are using:
wd s/= ws
wd s%= ws
For ALU movs instruction I just made up:
rd s= (i8) rs
rd s= (i16) rs
rd s= (i32) rs
For ALU32 movs I just made up:
wd s= (i8) ws
wd s= (i16) ws
wd s= (i32) ws
Thanks!
Quoted 11 lines
Hi Yonghong.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offsetI am adding the V4 BPF instructions to binutils. Where is the precise "pseudo-c" syntax used by the new instructions documented?
I looked at the tests in https://reviews.llvm.org/D144829 and:
Quoted 4 lines
For ALU sdiv/smod we are using:
rd s/= rs
rd s%= rs
Looks like I chose wisely, just by chance 8-)
Quoted 5 lines
For ALU movs instruction I just made up:
rd s= (i8) rs rd s= (i16) rs rd s= (i32) rs
Just changed that in binutils [1] to
rd = (s8) rs
rd = (s16) rs
rd = (s32) rs
Quoted 5 lines
For ALU32 movs I just made up:
wd s= (i8) ws wd s= (i16) ws wd s= (i32) ws
Just changed that in binutils [1] to
wd = (s8) ws
wd = (s16) ws
wd = (s32) ws
[1] https://sourceware.org/pipermail/binutils/2023-July/128544.html
Quoted 39 lines
Quoted 11 lines
Hi Yonghong.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offsetI am adding the V4 BPF instructions to binutils. Where is the precise "pseudo-c" syntax used by the new instructions documented?
I looked at the tests in https://reviews.llvm.org/D144829 and:
Quoted 4 lines
For ALU sdiv/smod we are using:
rd s/= rs
rd s%= rsLooks like I chose wisely, just by chance 8-)
Quoted 5 lines
For ALU movs instruction I just made up:
rd s= (i8) rs rd s= (i16) rs rd s= (i32) rsJust changed that in binutils [1] to
rd = (s8) rs
rd = (s16) rs
rd = (s32) rsQuoted 5 lines
For ALU32 movs I just made up:
wd s= (i8) ws wd s= (i16) ws wd s= (i32) ws
Quoted 7 lines
Just changed that in binutils [1] to
wd = (s8) ws
wd = (s16) ws
wd = (s32) ws[1] https://sourceware.org/pipermail/binutils/2023-July/128544.html
And finally for byte swap instructions:
rd = bswap16 rd
rd = bswap32 rd
rd = bswap64 rd
https://sourceware.org/pipermail/binutils/2023-July/128546.html
So, at this point we should have support for all the new BPF V4 instructions in the binutils opcodes, assembler and disassembler.
We are working now in getting GCC making good use of them. Salud!
Quoted 53 lines · Jose E. Marchesi
Quoted 39 lines
Quoted 11 lines
Hi Yonghong.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
. sign extended load
. sign extended mov
. bswap
. signed div/mod
. ja with 32-bit offsetI am adding the V4 BPF instructions to binutils. Where is the precise "pseudo-c" syntax used by the new instructions documented?
I looked at the tests in https://reviews.llvm.org/D144829 and:
Quoted 4 lines
For ALU sdiv/smod we are using:
rd s/= rs
rd s%= rsLooks like I chose wisely, just by chance 8-)
Quoted 5 lines
For ALU movs instruction I just made up:
rd s= (i8) rs rd s= (i16) rs rd s= (i32) rsJust changed that in binutils [1] to
rd = (s8) rs rd = (s16) rs rd = (s32) rsQuoted 5 lines
For ALU32 movs I just made up:
wd s= (i8) ws wd s= (i16) ws wd s= (i32) wsQuoted 7 lines
Just changed that in binutils [1] to
wd = (s8) ws wd = (s16) ws wd = (s32) ws[1] https://sourceware.org/pipermail/binutils/2023-July/128544.html
And finally for byte swap instructions:
rd = bswap16 rd
rd = bswap32 rd
rd = bswap64 rd
Quoted 6 lines
https://sourceware.org/pipermail/binutils/2023-July/128546.html
So, at this point we should have support for all the new BPF V4 instructions in the binutils opcodes, assembler and disassembler.
We are working now in getting GCC making good use of them.
Sounds great. Thanks!
I plan to push cpu v4 to llvm18 which should be available in the next week.
Quoted 1 line
Salud!