bpf, arm64: Support stack arguments
Subsystems (5)
Puranjay Mohan
maintainerv2 cover bpf, arm64: Support stack arguments
Hi Yonghong,
Please take these three patches into your next version of the stack arguments support. I have fixed the arm64 support patch so it works correctly with private stack.
Changelog:
v1: https://lore.kernel.org/all/[email protected]/
Changes in v2:
- Don't add ctx->stack_arg_size to off_adj when private stack is used. ([email protected])
This set add support of stack arguments to the arm64 JIT based on the preparatory work and x86 support in [1]. Arm64 allows passing 8 arguments in registers x0-x7 and remaining ones on stack. Currently, BPF supports passing 5 arguments all of which map to x0-x4.
BPF passes arguments in R1-R5 which map to x0-x4 on arm64, BPF reg R0 is mapped to arm64 reg x7, but as arm64 needs this register to pass arguments now, the first patch changes the mapping of BPF reg R0 to x8 which allows passing arguments in x0-x7 (8 arguments). 9+ arguments are passed on the stack.
All selftest pass:
./test_progs -t stack_arg,stack_arg_fail,stack_arg_kfunc,verifier_stack_arg
#431/1 stack_arg/global_many_args:OK #431/2 stack_arg/async_cb_many_args:OK #431/3 stack_arg/bpf2bpf:OK #431/4 stack_arg/kfunc:OK #431 stack_arg:OK #432/1 stack_arg_fail/test_stack_arg_big:OK #432/2 stack_arg_fail/r11 in ALU instruction:OK #432/3 stack_arg_fail/r11 store with non-DW size:OK #432/4 stack_arg_fail/r11 store with unaligned offset:OK #432/5 stack_arg_fail/r11 store with positive offset:OK #432/6 stack_arg_fail/r11 load with negative offset:OK #432/7 stack_arg_fail/r11 load with non-DW size:OK #432/8 stack_arg_fail/r11 store with zero offset:OK #432 stack_arg_fail:OK #631/1 verifier_stack_arg/stack_arg: subprog with 6 args:OK #631/2 verifier_stack_arg/stack_arg: two subprogs with >5 args:OK #631/3 verifier_stack_arg/stack_arg: read from uninitialized stack arg slot:OK #631/4 verifier_stack_arg/stack_arg: gap at offset -8, only wrote -16:OK #631/5 verifier_stack_arg/stack_arg: pruning with different stack arg types:OK #631/6 verifier_stack_arg/stack_arg: release_reference invalidates stack arg slot:OK #631/7 verifier_stack_arg/stack_arg: pkt pointer in stack arg slot invalidated after pull_data:OK #631/8 verifier_stack_arg/stack_arg: null propagation rejects deref on null branch:OK #631/9 verifier_stack_arg/stack_arg: missing store on one branch:OK #631/10 verifier_stack_arg/stack_arg: share a store for both branches:OK #631/11 verifier_stack_arg/stack_arg: write beyond max outgoing depth:OK #631/12 verifier_stack_arg/stack_arg: sequential calls reuse slots:OK #631 verifier_stack_arg:OK
Summary: 3/24 PASSED, 0 SKIPPED, 0 FAILED
Puranjay Mohan (3):
bpf, arm64: Map BPF_REG_0 to x8 instead of x7 bpf, arm64: Add JIT support for stack arguments selftests/bpf: Enable stack argument tests for arm64
arch/arm64/net/bpf_jit_comp.c | 91 ++++++++++++++++++- arch/arm64/net/bpf_timed_may_goto.S | 8 +- .../bpf/progs/btf__stack_arg_precision.c | 3 +- tools/testing/selftests/bpf/progs/stack_arg.c | 3 +- .../selftests/bpf/progs/stack_arg_kfunc.c | 3 +- .../selftests/bpf/progs/stack_arg_precision.c | 3 +- .../selftests/bpf/progs/verifier_jit_inline.c | 2 +- .../selftests/bpf/progs/verifier_ldsx.c | 6 +- .../bpf/progs/verifier_private_stack.c | 10 +- .../selftests/bpf/progs/verifier_stack_arg.c | 3 +- 10 files changed, 111 insertions(+), 21 deletions(-)
Signature · 1 line
2.52.0
Puranjay Mohan
maintainerPATCH v2 1/3 bpf, arm64: Map BPF_REG_0 to x8 instead of x7
Move the BPF return value register from x7 to x8, freeing x7 for use as an argument register. AAPCS64 designates x8 as the indirect result location register; it is caller-saved and not used for argument passing, making it a suitable home for BPF_REG_0.
This is a prerequisite for stack argument support, which needs x5-x7 to pass arguments 6-8 to native kfuncs following the AAPCS64 calling convention.
--- arch/arm64/net/bpf_jit_comp.c | 4 ++-- arch/arm64/net/bpf_timed_may_goto.S | 8 ++++---- .../testing/selftests/bpf/progs/verifier_jit_inline.c | 2 +- tools/testing/selftests/bpf/progs/verifier_ldsx.c | 6 +++--- .../selftests/bpf/progs/verifier_private_stack.c | 10 +++++----- 5 files changed, 15 insertions(+), 15 deletions(-)
arch/arm64/net/bpf_jit_comp.c +2 -2
@@ -47,7 +47,7 @@/* Map BPF registers to A64 registers */ static const int bpf2a64[] = {/* return value from in-kernel function, and exit value from eBPF */ - [BPF_REG_0 ] =A64_R (7 ), + [BPF_REG_0 ] =A64_R (8 ),/* arguments from eBPF program to in-kernel function */ [BPF_REG_1 ] =A64_R (0 ), [BPF_REG_2 ] =A64_R (1 ),
@@ -1048,7 +1048,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)/* Restore FP/LR registers */ emit (A64_POP (A64_FP ,A64_LR ,A64_SP ), ctx); -/* Move the return value from bpf:r0 (aka x7) to x0 */ +/* Move the return value from bpf:r0 (aka x8) to x0 */ emit (A64_MOV (1 ,A64_R (0 ), r0), ctx);/* Authenticate lr */
arch/arm64/net/bpf_timed_may_goto.S +4 -4
@@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) stp x29, x30, [sp, #-64]! mov x29, sp - /* Save BPF registers R0 - R5 (x7, x0-x4)*/ - stp x7, x0, [sp, #16] + /* Save BPF registers R0 - R5 (x8, x0-x4)*/ + stp x8, x0, [sp, #16] stp x1, x2, [sp, #32] stp x3, x4, [sp, #48]
@@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */ mov x9, x0 - /* Restore BPF registers R0 - R5 (x7, x0-x4) */ - ldp x7, x0, [sp, #16] + /* Restore BPF registers R0 - R5 (x8, x0-x4) */ + ldp x8, x0, [sp, #16] ldp x1, x2, [sp, #32] ldp x3, x4, [sp, #48]
tools/testing/selftests/bpf/progs/verifier_jit_inline.c +1 -1
@@ -9,7 +9,7 @@ __success __retval(0) __arch_x86_64 __jited(" addq %gs:{{.*}}, %rax") __arch_arm64 -__jited(" mrs x7, SP_EL0") +__jited(" mrs x8, SP_EL0")int inline_bpf_get_current_task (void ) {bpf_get_current_task ();
tools/testing/selftests/bpf/progs/verifier_ldsx.c +3 -3
@@ -274,11 +274,11 @@ __jited("movslq 0x10(%rdi,%r12), %r15") __jited("movswq 0x18(%rdi,%r12), %r15") __jited("movsbq 0x20(%rdi,%r12), %r15") __arch_arm64 -__jited("add x11, x7, x28") +__jited("add x11, x8, x28") __jited("ldrsw x21, [x11, #0x10]") -__jited("add x11, x7, x28") +__jited("add x11, x8, x28") __jited("ldrsh x21, [x11, #0x18]") -__jited("add x11, x7, x28") +__jited("add x11, x8, x28") __jited("ldrsb x21, [x11, #0x20]") __jited("add x11, x0, x28")__jited ("ldrsw x22, [x11, #0x10]" )
tools/testing/selftests/bpf/progs/verifier_private_stack.c +5 -5
@@ -170,12 +170,12 @@ __jited(" mrs x10, TPIDR_EL{{[0-1]}}") __jited(" add x27, x27, x10") __jited(" add x25, x27, {{.*}}") __jited(" bl 0x{{.*}}") -__jited(" mov x7, x0") +__jited(" mov x8, x0") __jited(" mov x0, #0x2a") __jited(" str x0, [x27]") __jited(" bl 0x{{.*}}") -__jited(" mov x7, x0") -__jited(" mov x7, #0x0") +__jited(" mov x8, x0") +__jited(" mov x8, #0x0") __jited(" ldp x25, x27, [sp], {{.*}}")__naked void private_stack_callback (void ) {
@@ -220,7 +220,7 @@ __jited(" mov x0, #0x2a") __jited(" str x0, [x27]") __jited(" mov x0, #0x0") __jited(" bl 0x{{.*}}") -__jited(" mov x7, x0") +__jited(" mov x8, x0") __jited(" ldp x27, x28, [sp], #0x10")int private_stack_exception_main_prog (void ) {
@@ -258,7 +258,7 @@ __jited(" add x25, x27, {{.*}}") __jited(" mov x0, #0x2a") __jited(" str x0, [x27]") __jited(" bl 0x{{.*}}") -__jited(" mov x7, x0") +__jited(" mov x8, x0") __jited(" ldp x27, x28, [sp], #0x10")int private_stack_exception_sub_prog (void ) {
Signature · 1 line
2.52.0
Re: [PATCH bpf-next v2 1/3] bpf, arm64: Map BPF_REG_0 to x8 instead of x7
Quoted 11 lines · Puranjay Mohan
Move the BPF return value register from x7 to x8, freeing x7 for use as an argument register. AAPCS64 designates x8 as the indirect result location register; it is caller-saved and not used for argument passing, making it a suitable home for
BPF_REG_0.This is a prerequisite for stack argument support, which needs x5-x7 to pass arguments 6-8 to native kfuncs following the AAPCS64 calling convention.
Signed-off-by Yonghong Song <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>
Quoted 7 lines
--- arch/arm64/net/bpf_jit_comp.c | 4 ++-- arch/arm64/net/bpf_timed_may_goto.S | 8 ++++---- .../testing/selftests/bpf/progs/verifier_jit_inline.c | 2 +- tools/testing/selftests/bpf/progs/verifier_ldsx.c | 6 +++--- .../selftests/bpf/progs/verifier_private_stack.c | 10 +++++----- 5 files changed, 15 insertions(+), 15 deletions(-)
Makes sense to me:
Will
Puranjay Mohan
maintainerPATCH v2 2/3 bpf, arm64: Add JIT support for stack arguments
In reply to Puranjay MohanImplement stack argument passing for BPF-to-BPF and kfunc calls with more than 5 parameters on arm64, following the AAPCS64 calling convention.
BPF R1-R5 already map to x0-x4. With BPF_REG_0 moved to x8 by the previous commit, x5-x7 are free for arguments 6-8. Arguments 9-12 spill onto the stack at [SP+0], [SP+8], ... and the callee reads them from [FP+16], [FP+24], ... (above the saved FP/LR pair).
BPF convention uses fixed offsets from BPF_REG_PARAMS (r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.
--- arch/arm64/net/bpf_jit_comp.c | 87 ++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)
arch/arm64/net/bpf_jit_comp.c +86 -1
@@ -86,6 +86,7 @@ struct jit_ctx {__le32 *image;__le32 *ro_image;u32 stack_size; +u16 stack_arg_size;u64 user_vm_start;u64 arena_vm_start;bool fp_used;
@@ -533,13 +534,19 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* | | * +-----+ <= (BPF_FP - prog->aux->stack_depth) * |RSVD | padding -* current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) +* +-----+ <= (BPF_FP - ctx->stack_size) +* | | +* | ... | outgoing stack args (9+, if any) +* | | +* current A64_SP => +-----+ * | | * | ... | Function call stack * | | * +-----+ * low * +* Stack args 6-8 are passed in x5-x7, args 9+ at [SP]. +* Incoming args 9+ are at [FP + 16], [FP + 24], ... */ emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);
@@ -613,6 +620,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)if (ctx->stack_size && !ctx->priv_sp_used)emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_size), ctx); +if (ctx->stack_arg_size) +emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_arg_size), ctx); +if (ctx->arena_vm_start)emit_a64_mov_i64 (arena_vm_base, ctx->arena_vm_start, ctx);
@@ -673,6 +683,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx)/* Update tail_call_cnt if the slot is populated. */ emit (A64_STR64I (tcc, ptr,0 ), ctx); +if (ctx->stack_arg_size) +emit (A64_ADD_I (1 ,A64_SP ,A64_SP , ctx->stack_arg_size), ctx); +/* restore SP */ if (ctx->stack_size && !ctx->priv_sp_used)emit (A64_ADD_I (1 ,A64_SP ,A64_SP , ctx->stack_size), ctx);
@@ -1034,6 +1047,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)const u8 r0 = bpf2a64[BPF_REG_0 ];const u8 ptr = bpf2a64[TCCNT_PTR ]; +if (ctx->stack_arg_size) +emit (A64_ADD_I (1 ,A64_SP ,A64_SP , ctx->stack_arg_size), ctx); +/* We're done with BPF stack */ if (ctx->stack_size && !ctx->priv_sp_used)emit (A64_ADD_I (1 ,A64_SP ,A64_SP , ctx->stack_size), ctx);
@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,return 0 ; } +static const u8 stack_arg_reg[] = {A64_R (5 ),A64_R (6 ),A64_R (7 ) }; + +#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg) + +static void emit_stack_arg_load (u8 dst,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , dst, stack_arg_reg[idx]), ctx); +else +emit (A64_LDR64I (dst,A64_FP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64) +16 ), ctx); +} + +static void emit_stack_arg_store (u8 src_a64,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = -bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], src_a64), ctx); +else +emit (A64_STR64I (src_a64,A64_SP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64)), ctx); +} + +static void emit_stack_arg_store_imm (s32 imm,s16 bpf_off,const u8 tmp,struct jit_ctx *ctx) +{ +int idx = -bpf_off /sizeof (u64) -1 ; + +emit_a64_mov_i (1 , tmp, imm, ctx); +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], tmp), ctx); +else +emit (A64_STR64I (tmp,A64_SP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64)), ctx); +} +/* JITs an eBPF instruction. * Returns: * 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1646,6 +1697,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insncase BPF_LDX |BPF_MEM |BPF_H :case BPF_LDX |BPF_MEM |BPF_B :case BPF_LDX |BPF_MEM |BPF_DW : +if (insn->src_reg ==BPF_REG_PARAMS ) { +emit_stack_arg_load (dst, off, ctx); +break ; + } + fallthrough;case BPF_LDX |BPF_PROBE_MEM |BPF_DW :case BPF_LDX |BPF_PROBE_MEM |BPF_W :case BPF_LDX |BPF_PROBE_MEM |BPF_H :
@@ -1672,6 +1728,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (src == fp) { src_adj = ctx->priv_sp_used ? priv_sp :A64_SP ; off_adj = off + ctx->stack_size; +if (!ctx->priv_sp_used) + off_adj += ctx->stack_arg_size; }else { src_adj = src; off_adj = off;
@@ -1752,6 +1810,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insncase BPF_ST |BPF_MEM |BPF_H :case BPF_ST |BPF_MEM |BPF_B :case BPF_ST |BPF_MEM |BPF_DW : +if (insn->dst_reg ==BPF_REG_PARAMS ) { +emit_stack_arg_store_imm (imm, off, tmp, ctx); +break ; + } + fallthrough;case BPF_ST |BPF_PROBE_MEM32 |BPF_B :case BPF_ST |BPF_PROBE_MEM32 |BPF_H :case BPF_ST |BPF_PROBE_MEM32 |BPF_W :
@@ -1763,6 +1826,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (dst == fp) { dst_adj = ctx->priv_sp_used ? priv_sp :A64_SP ; off_adj = off + ctx->stack_size; +if (!ctx->priv_sp_used) + off_adj += ctx->stack_arg_size; }else { dst_adj = dst; off_adj = off;
@@ -1814,6 +1879,11 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insncase BPF_STX |BPF_MEM |BPF_H :case BPF_STX |BPF_MEM |BPF_B :case BPF_STX |BPF_MEM |BPF_DW : +if (insn->dst_reg ==BPF_REG_PARAMS ) { +emit_stack_arg_store (src, off, ctx); +break ; + } + fallthrough;case BPF_STX |BPF_PROBE_MEM32 |BPF_B :case BPF_STX |BPF_PROBE_MEM32 |BPF_H :case BPF_STX |BPF_PROBE_MEM32 |BPF_W :
@@ -1825,6 +1895,8 @@ static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insnif (dst == fp) { dst_adj = ctx->priv_sp_used ? priv_sp :A64_SP ; off_adj = off + ctx->stack_size; +if (!ctx->priv_sp_used) + off_adj += ctx->stack_arg_size; }else { dst_adj = dst; off_adj = off;
@@ -2065,6 +2137,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx.user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx.arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); +if (prog->aux->stack_arg_depth > prog->aux->incoming_stack_arg_depth) { +u16 outgoing = prog->aux->stack_arg_depth - prog->aux->incoming_stack_arg_depth; +int nr_on_stack = outgoing /sizeof (u64) -NR_STACK_ARG_REGS ; + +if (nr_on_stack >0 ) + ctx.stack_arg_size =round_up (nr_on_stack *sizeof (u64),16 ); + } +if (priv_stack_ptr) ctx.priv_sp_used = true;
@@ -2229,6 +2309,11 @@ bool bpf_jit_supports_kfunc_call(void)return true; } +bool bpf_jit_supports_stack_args (void ) +{ +return true; +} +void *bpf_arch_text_copy (void *dst,void *src,size_t len) {if (!aarch64_insn_copy (dst, src, len))
Signature · 1 line
2.52.0
Will Deacon
maintainerRe: [PATCH bpf-next v2 2/3] bpf, arm64: Add JIT support for stack arguments
Quoted 8 lines · Puranjay Mohan
Implement stack argument passing for BPF-to-BPF and kfunc calls with more than 5 parameters on arm64, following the AAPCS64 calling convention.
BPF R1-R5 already map to x0-x4. With
BPF_REG_0moved to x8 by the previous commit, x5-x7 are free for arguments 6-8. Arguments 9-12 spill onto the stack at [SP+0], [SP+8], ... and the callee reads them from [FP+16], [FP+24], ... (above the saved FP/LR pair).
How does that work with kfuncs? I think the PCS means that they will expect to pick the stack arguments starting at [SP+0]. Or are you saying that SP == FP+16 on entry to the callee? It's hard to reconcile that with the ASCII art in build_prologue() because neither "current A64_FP" nor "BPF_FP" point below A64_SP and it's not clear which of them you're referring to when you refer to "FP" on its own.
Quoted 32 lines
BPF convention uses fixed offsets from
BPF_REG_PARAMS(r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.Signed-off-by Yonghong Song <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- arch/arm64/net/bpf_jit_comp.c | 87 ++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)arch/arm64/net/bpf_jit_comp.c +5
-1@@ -86,6 +86,7 @@ struct jit_ctx {__le32 *image;__le32 *ro_image;u32 stack_size; +u16 stack_arg_size;u64 user_vm_start;u64 arena_vm_start;bool fp_used;@@ -533,13 +534,19 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* | | * +-----+ <= (BPF_FP - prog->aux->stack_depth) * |RSVD | padding -* current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) +* +-----+ <= (BPF_FP - ctx->stack_size) +* | | +* | ... | outgoing stack args (9+, if any) +* | |
arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 550
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +3
-0+* current A64_SP => +-----+ * | | * | ... | Function call stack * | | * +-----+ * low * +* Stack args 6-8 are passed in x5-x7, args 9+ at [SP]. +* Incoming args 9+ are at [FP + 16], [FP + 24], ... */
I assume the arguments being passed are all <= 64-bit scalar types? If we ever want to pass anything with > 64-bit alignment or to a varargs function, then the rules for allocation get a little hairy.
arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 624
Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c +2
-0emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);@@ -613,6 +620,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)if (ctx->stack_size && !ctx->priv_sp_used)emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_size), ctx); +if (ctx->stack_arg_size) +emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_arg_size), ctx);
How do you ensure that the stack pointer is always 16-byte aligned? We run with SP alignment checking enabled, so you need to take care of that.
Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c +3
-0@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,return 0 ; } +static const u8 stack_arg_reg[] = {A64_R (5 ),A64_R (6 ),A64_R (7 ) }; + +#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1222
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10
-0+ +static void emit_stack_arg_load (u8 dst,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , dst, stack_arg_reg[idx]), ctx); +else +emit (A64_LDR64I (dst,A64_FP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64) +16 ), ctx); +}
Is it worth asserting that bpf_off >= 8 here or can we rely on that? I struggled to find any details about how bpf passes arguments on the stack (beyond what you describe in the commit message) and grepping for BPF_REG_PARAMS didn't help either.
Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c +7
-0+static void emit_stack_arg_store (u8 src_a64,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = -bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], src_a64), ctx); +else
arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1240
Quoted 10 lines
arch/arm64/net/bpf_jit_comp.c +10
-0+emit (A64_STR64I (src_a64,A64_SP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64)), ctx); +} + +static void emit_stack_arg_store_imm (s32 imm,s16 bpf_off,const u8 tmp,struct jit_ctx *ctx) +{ +int idx = -bpf_off /sizeof (u64) -1 ; + +emit_a64_mov_i (1 , tmp, imm, ctx); +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], tmp), ctx);
nit: You seem to have redundant MOVs here.
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 2146
Quoted 11 lines
arch/arm64/net/bpf_jit_comp.c +7
-0@@ -2065,6 +2137,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx.user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx.arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); +if (prog->aux->stack_arg_depth > prog->aux->incoming_stack_arg_depth) { +u16 outgoing = prog->aux->stack_arg_depth - prog->aux->incoming_stack_arg_depth; +int nr_on_stack = outgoing /sizeof (u64) -NR_STACK_ARG_REGS ; + +if (nr_on_stack >0 ) + ctx.stack_arg_size =round_up (nr_on_stack *sizeof (u64),16 ); + }
ah, that's presumably where you handle the SP alignment. I think a comment would really help folks here... In fact, how does this interact with the same sort of SP adjustment that already exists in build_prologue()? Can we avoid the pointless re-alignment?
Will
Re: [PATCH bpf-next v2 2/3] bpf, arm64: Add JIT support for stack arguments
Hi Will,
Thanks for your review.
These patches have already landed in bpf-next tree as a part of this
series: https://lore.kernel.org/all/[email protected]/
I will take all your feedback and send fixes for all the required changes.
Quoted 6 lines
How does that work with kfuncs? I think the PCS means that they will expect to pick the stack arguments starting at [SP+0]. Or are you saying that SP == FP+16 on entry to the callee? It's hard to reconcile that with the ASCII art in
build_prologue()because neither "currentA64_FP" nor "BPF_FP" point belowA64_SPand it's not clear which of them you're referring to when you refer to "FP" on its own.
At entry to kfunc the arguments are starting at [SP+0] like a normal C callee function would expect.
The line "callee reads them from [FP+16], [FP+24]" is for a BPF callee which always emits a frame record.
So the caller puts the arguments at SP+0, SP+8, .. and then the callee creates the frame record in the prologue which means the first argument is at FP+16, ....
This is just an implementation detail about how the JITed BPF callee programs can read stack arguments from FP+16, ... easily.
Quoted 39 lines
Quoted 39 lines
BPF convention uses fixed offsets from
BPF_REG_PARAMS(r11): off=-8 is always arg 6, off=-16 arg 7, etc. The verifier invalidates all outgoing stack arg slots after each call, so the compiler must re-store before every call. This means x5-x7 don't need to be saved on stack.Signed-off-by Yonghong Song <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>--- arch/arm64/net/bpf_jit_comp.c | 87 ++++++++++++++++++++++++++++++++++- 1 file changed, 86 insertions(+), 1 deletion(-)arch/arm64/net/bpf_jit_comp.c +6
-1@@ -86,6 +86,7 @@ struct jit_ctx {__le32 *image;__le32 *ro_image;u32 stack_size; +u16 stack_arg_size;u64 user_vm_start;u64 arena_vm_start;bool fp_used;@@ -533,13 +534,19 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)* | | * +-----+ <= (BPF_FP - prog->aux->stack_depth) * |RSVD | padding -* current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) +* +-----+ <= (BPF_FP - ctx->stack_size) +* | | +* | ... | outgoing stack args (9+, if any) +* | | +* current A64_SP => +-----+ * | | * | ... | Function call stack * | | * +-----+ * low
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 550
Quoted 4 lines
arch/arm64/net/bpf_jit_comp.c +2
-0* +* Stack args 6-8 are passed in x5-x7, args 9+ at [SP]. +* Incoming args 9+ are at [FP + 16], [FP + 24], ... */ I assume the arguments being passed are all <= 64-bit scalar types? If we ever want to pass anything with > 64-bit alignment or to a varargs function, then the rules for allocation get a little hairy.
Yes, right now only 64-bit scalars are supported.
Quoted 11 lines
arch/arm64/net/bpf_jit_comp.c static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · line 624
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c +2
-0emit_kcfi (is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx);@@ -613,6 +620,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf)if (ctx->stack_size && !ctx->priv_sp_used)emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_size), ctx); +if (ctx->stack_arg_size) +emit (A64_SUB_I (1 ,A64_SP ,A64_SP , ctx->stack_arg_size), ctx);How do you ensure that the stack pointer is always 16-byte aligned? We run with SP alignment checking enabled, so you need to take care of that.
Down below as you have already found, stack_arg_size is always aligned.
Quoted 15 lines
Quoted 15 lines
arch/arm64/net/bpf_jit_comp.c +10
-0@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,return 0 ; } +static const u8 stack_arg_reg[] = {A64_R (5 ),A64_R (6 ),A64_R (7 ) }; + +#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg) + +static void emit_stack_arg_load (u8 dst,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , dst, stack_arg_reg[idx]), ctx);
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1222
Quoted 3 lines
arch/arm64/net/bpf_jit_comp.c +3
-0+else +emit (A64_LDR64I (dst,A64_FP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64) +16 ), ctx); +}Is it worth asserting that
bpf_off>= 8 here or can we rely on that? I struggled to find any details about how bpf passes arguments on the stack (beyond what you describe in the commit message) and grepping forBPF_REG_PARAMSdidn't help either.
This and other patches in this set has more information about this:
https://lore.kernel.org/all/[email protected]/
Quoted 12 lines
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c +11
-0+static void emit_stack_arg_store (u8 src_a64,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = -bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], src_a64), ctx); +else +emit (A64_STR64I (src_a64,A64_SP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64)), ctx); +} + +static void emit_stack_arg_store_imm (s32 imm,s16 bpf_off,const u8 tmp,struct jit_ctx *ctx)
Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1240
Quoted 6 lines
arch/arm64/net/bpf_jit_comp.c +6
-0+{ +int idx = -bpf_off /sizeof (u64) -1 ; + +emit_a64_mov_i (1 , tmp, imm, ctx); +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , stack_arg_reg[idx], tmp), ctx);nit: You seem to have redundant MOVs here.
Ack, will fix this.
Quoted 5 lines
Quoted 5 lines
arch/arm64/net/bpf_jit_comp.c +0
-0@@ -2065,6 +2137,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr ctx.user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx.arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena);
Quoted 12 lines
arch/arm64/net/bpf_jit_comp.c struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · line 2146
Quoted 7 lines
arch/arm64/net/bpf_jit_comp.c +7
-0+if (prog->aux->stack_arg_depth > prog->aux->incoming_stack_arg_depth) { +u16 outgoing = prog->aux->stack_arg_depth - prog->aux->incoming_stack_arg_depth; +int nr_on_stack = outgoing /sizeof (u64) -NR_STACK_ARG_REGS ; + +if (nr_on_stack >0 ) + ctx.stack_arg_size =round_up (nr_on_stack *sizeof (u64),16 ); + }ah, that's presumably where you handle the SP alignment. I think a comment would really help folks here... In fact, how does this interact with the same sort of SP adjustment that already exists in
build_prologue()? Can we avoid the pointless re-alignment?
Yeah, you are right, if ctx->stack_size + ctx-> stack_arg_size is aligned then we don't need to separately align these two, I can move this from here and consolidate it in the build_prologue() function and also add a comment there.
Quoted 1 line
Will
Will Deacon
maintainerRe: [PATCH bpf-next v2 2/3] bpf, arm64: Add JIT support for stack arguments
Quoted 5 lines · Puranjay Mohan
Thanks for your review.
These patches have already landed in bpf-next tree as a part of this
series: https://lore.kernel.org/all/[email protected]/I will take all your feedback and send fixes for all the required changes.
Thanks. My review backlog is huge, but I still try to look at stuff even after it's been merged.
Quoted 10 lines
Quoted 6 lines
How does that work with kfuncs? I think the PCS means that they will expect to pick the stack arguments starting at [SP+0]. Or are you saying that SP == FP+16 on entry to the callee? It's hard to reconcile that with the ASCII art in
build_prologue()because neither "currentA64_FP" nor "BPF_FP" point belowA64_SPand it's not clear which of them you're referring to when you refer to "FP" on its own.At entry to kfunc the arguments are starting at [SP+0] like a normal C callee function would expect.
Quoted 7 lines
The line "callee reads them from [FP+16], [FP+24]" is for a BPF callee which always emits a frame record.
So the caller puts the arguments at SP+0, SP+8, .. and then the callee creates the frame record in the prologue which means the first argument is at FP+16, ....
This is just an implementation detail about how the JITed BPF callee programs can read stack arguments from FP+16, ... easily.
That makes sense, but if you're able to rejig the diagram in build_prologue() to mirror that then it would really help us to remember it in future.
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
arch/arm64/net/bpf_jit_comp.c +10
-0@@ -1191,6 +1207,41 @@ static int add_exception_handler(const struct bpf_insn *insn,return 0 ; } +static const u8 stack_arg_reg[] = {A64_R (5 ),A64_R (6 ),A64_R (7 ) }; + +#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg) + +static void emit_stack_arg_load (u8 dst,s16 bpf_off,struct jit_ctx *ctx) +{ +int idx = bpf_off /sizeof (u64) -1 ; + +if (idx <NR_STACK_ARG_REGS ) +emit (A64_MOV (1 , dst, stack_arg_reg[idx]), ctx);Quoted 8 lines
arch/arm64/net/bpf_jit_comp.c static int add_exception_handler(const struct bpf_insn *insn, · line 1222
Quoted 3 lines
arch/arm64/net/bpf_jit_comp.c +3
-0+else +emit (A64_LDR64I (dst,A64_FP , (idx -NR_STACK_ARG_REGS ) *sizeof (u64) +16 ), ctx); +}Is it worth asserting that
bpf_off>= 8 here or can we rely on that? I struggled to find any details about how bpf passes arguments on the stack (beyond what you describe in the commit message) and grepping forBPF_REG_PARAMSdidn't help either.
Quoted 2 lines
This and other patches in this set has more information about this:
https://lore.kernel.org/all/[email protected]/
Thanks, that matches what I'd managed to figure out.
Will
Puranjay Mohan
maintainerPATCH v2 3/3 selftests/bpf: Enable stack argument tests for arm64
In reply to Puranjay MohanNow that arm64 supports stack arguments, enable the existing stack_arg, stack_arg_kfunc and verifier_stack_arg tests for __TARGET_ARCH_arm64.
--- tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_kfunc.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 3 ++- 5 files changed, 10 insertions(+), 5 deletions(-)
tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c +2 -1
@@ -4,7 +4,8 @@#include <bpf/bpf_helpers.h> #include "../test_kmods/bpf_testmod_kfunc.h" -#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT) +#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)long subprog_call_mem_kfunc (long a,long b,long c,long d,long e,long size) {
tools/testing/selftests/bpf/progs/stack_arg.c +2 -1
@@ -21,7 +21,8 @@ struct {int timer_result; -#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT) +#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)const volatile bool has_stack_arg = true;
tools/testing/selftests/bpf/progs/stack_arg_kfunc.c +2 -1
@@ -6,7 +6,8 @@#include "bpf_kfuncs.h" #include "../test_kmods/bpf_testmod_kfunc.h" -#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT) +#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)const volatile bool has_stack_arg = true;
tools/testing/selftests/bpf/progs/stack_arg_precision.c +2 -1
@@ -6,7 +6,8 @@#include "../test_kmods/bpf_testmod_kfunc.h" #include "bpf_misc.h" -#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT) +#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)/* Force kfunc extern BTF generation for inline asm call below. * Uses its own SEC so it's not included as a .text subprog.
tools/testing/selftests/bpf/progs/verifier_stack_arg.c +2 -1
@@ -12,7 +12,8 @@ struct { __type(value, long long); } map_hash_8b SEC(".maps"); -#if defined(__TARGET_ARCH_x86) && defined(__BPF_FEATURE_STACK_ARGUMENT) +#if (defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)) && \ + defined(__BPF_FEATURE_STACK_ARGUMENT)__noinline __used static int subprog_6args(int a, int b, int c, int d, int e, int f)
Signature · 1 line
2.52.0
Will Deacon
maintainerRe: [PATCH bpf-next v2 3/3] selftests/bpf: Enable stack argument tests for arm64
Quoted 5 lines · Puranjay Mohan
Now that arm64 supports stack arguments, enable the existing
stack_arg,stack_arg_kfuncandverifier_stack_argtests for__TARGET_ARCH_arm64.Signed-off-by Yonghong Song <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>
Quoted 7 lines
--- tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_kfunc.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 3 ++- 5 files changed, 10 insertions(+), 5 deletions(-)
Do any of these tests pass more than eight parameters? If not, then they're not touching any of the stack stuff you added for arm64.
Will
Re: [PATCH bpf-next v2 3/3] selftests/bpf: Enable stack argument tests for arm64
Quoted 15 lines · Will Deacon
Quoted 5 lines · Puranjay Mohan
Now that arm64 supports stack arguments, enable the existing
stack_arg,stack_arg_kfuncandverifier_stack_argtests for__TARGET_ARCH_arm64.Signed-off-by Yonghong Song <[email protected]>Signed-off-by Puranjay Mohan <[email protected]>Quoted 7 lines
--- tools/testing/selftests/bpf/progs/btf__stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_kfunc.c | 3 ++- tools/testing/selftests/bpf/progs/stack_arg_precision.c | 3 ++- tools/testing/selftests/bpf/progs/verifier_stack_arg.c | 3 ++- 5 files changed, 10 insertions(+), 5 deletions(-)
Quoted 2 lines
Do any of these tests pass more than eight parameters? If not, then they're not touching any of the stack stuff you added for arm64.
The test "test_two_callees" has ret += foo2(a, b, c, d, e, a + 5, a + 6, a + 7, a + 8, a + 9);
so, 10 arguments are tested.
But you are right, I will send a patch to increase all these tests to pass 10 arguments so it increases test coverage on arm64.
Thanks,
Puranjay