bpf, x64: Introduce two tailcall enhancements
Subsystems (4)
cover bpf, x64: Introduce two tailcall enhancements
This patch set introduces two enhancements aimed at improving tailcall handling in the x64 JIT:
1. Tailcall info is propagated only for subprogs.
2. Tailcall info is propagated through the trampoline only when the target is a subprog and it is tail_call_reachable.
v1 -> v2:
* Address comment from Alexei:
* Rather live with tail call inefficiency than abuse insns fields further.
Leon Hwang (2):
bpf, x64: Propagate tailcall info only for subprogs
bpf, verifier: Check trampoline target is tail_call_reachable subprog
arch/x86/net/bpf_jit_comp.c | 3 ++- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 4 +++- 3 files changed, 6 insertions(+), 2 deletions(-)
Signature · 1 line
2.44.0
PATCH 1/2 bpf, x64: Propagate tailcall info only for subprogs
In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.
--- arch/x86/net/bpf_jit_comp.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +2 -1
@@ -2124,10 +2124,11 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }
Signature · 1 line
2.44.0
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for subprogs
Quoted 4 lines · Leon Hwang
In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.
Signed-off-by Leon Hwang <[email protected]>
PATCH 2/2 bpf, verifier: Check trampoline target is tail_call_reachable subprog
In reply to Leon HwangIn the x86_64 JIT, tailcall info is propagated through the trampoline when the target program is tail_call_reachable. However, this propagation is unnecessary if the target is a main prog, or a subprog that is not tail_call_reachable.
Since the verifier can determine if a subprog is tail_call_reachable, it should only propagate tailcall info when the target is subprog and the subprog is actually tail_call_reachable.
--- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/bpf.h +1 -0
@@ -1253,6 +1253,7 @@ struct bpf_attach_target_info {struct module *tgt_mod;const char *tgt_name;const struct btf_type *tgt_type; +bool tgt_tail_call_reachable; };#define BPF_DISPATCHER_MAX 48/* Fits in 2048B */
kernel/bpf/verifier.c +3 -1
@@ -21946,6 +21946,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,bpf_log (log,"Subprog %s doesn't exist\n" , tname);return -EINVAL ; } + tgt_info->tgt_tail_call_reachable = subprog && + aux->func[subprog]->aux->tail_call_reachable;if (aux->func && aux->func[subprog]->aux->exception_cb) { bpf_log(log, "%s programs cannot attach to exception callback\n",
@@ -22315,7 +22317,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)if (!tr)return -ENOMEM ; -if (tgt_prog && tgt_prog->aux->tail_call_reachable) +if (tgt_prog && tgt_info.tgt_tail_call_reachable) tr->flags =BPF_TRAMP_F_TAIL_CALL_CTX ; prog->aux->dst_trampoline = tr;
Signature · 1 line
2.44.0