bpf, x64: Introduce two tailcall enhancements
Subsystems (4)
cover bpf, x64: Introduce two tailcall enhancements
This patch set introduces two enhancements aimed at improving tailcall handling in the x64 JIT:
1. Tailcall info is propagated to a subprog only if the subprog is tail_call_reachable.
2. Tailcall info is propagated through the trampoline only when the target is a subprog and it is tail_call_reachable.
Leon Hwang (2):
bpf, x64: Propagate tailcall info only for tail_call_reachable
subprogs
bpf, verifier: Check trampoline attach target is tail_call_reachable
subprog
arch/x86/net/bpf_jit_comp.c | 4 +++- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 10 +++++++++- 3 files changed, 13 insertions(+), 2 deletions(-)
Signature · 1 line
2.44.0
PATCH 1/2 bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
In the x86_64 JIT, when calling a function, tailcall info is propagated if the program is tail_call_reachable, regardless of whether the function is a subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.
The verifier can determine if a subprog is tail_call_reachable. Therefore, it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actually tail_call_reachable.
--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +3 -1
@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ; +bool subprog_tail_call_reachable = dst_reg;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && subprog_tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }
kernel/bpf/verifier.c +6 -0
@@ -19990,6 +19990,12 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn[0 ].imm = (u32 )addr; insn[1 ].imm = addr >>32 ; } + +if (bpf_pseudo_call (insn)) +/* In the x86_64 JIT, tailcall information can only be +* propagated if the subprog is tail_call_reachable. +*/ + insn->dst_reg = env->subprog_info[subprog].tail_call_reachable; } err =bpf_prog_alloc_jited_linfo (prog);
Signature · 1 line
2.44.0
Yonghong Song
reviewerRe: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 20 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if the program istail_call_reachable, regardless of whether the function is a subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore, it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +0
-0
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2132
Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ; +bool subprog_tail_call_reachable = dst_reg;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && subprog_tail_call_reachable) {
Why we need subprog_tail_call_reachable? Does
tail_call_reachable && psueudo_call
work the same way?
Quoted 20 lines
arch/x86/net/bpf_jit_comp.c +0
-0LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }kernel/bpf/verifier.c +6
-0@@ -19990,6 +19990,12 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn[0 ].imm = (u32 )addr; insn[1 ].imm = addr >>32 ; } + +if (bpf_pseudo_call (insn)) +/* In the x86_64 JIT, tailcall information can only be +* propagated if the subprog is tail_call_reachable. +*/ + insn->dst_reg = env->subprog_info[subprog].tail_call_reachable; } err =bpf_prog_alloc_jited_linfo (prog);
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 29 lines · Yonghong Song
Quoted 27 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if
the program istail_call_reachable, regardless of whether the function is a
subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore,
it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>---
arch/x86/net/bpf_jit_comp.c| 4 +++-
kernel/bpf/verifier.c | 6 ++++++2 files changed, 9 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +2
-0@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { + bool pseudo_call = src_reg == BPF_PSEUDO_CALL; + bool subprog_tail_call_reachable = dst_reg;
Quoted 9 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2132
Quoted 4 lines
arch/x86/net/bpf_jit_comp.c +1
-1u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + if (pseudo_call && subprog_tail_call_reachable) {Why we need
subprog_tail_call_reachable? Does
tail_call_reachable &&psueudo_call
work the same way?
'tail_call_reachable && pseudo_call' works too. However, it will propagate tailcall info to subprog even if the subprog is not tail_call_reachable.
subprog_tail_call_reachable indicates the subprog requires tailcall info from its caller.
So, 'pseudo_call && subprog_tail_call_reachable' is better.
Thanks,
Leon
Yonghong Song
reviewerRe: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 38 lines · Leon Hwang
Quoted 19 lines · Yonghong Song
Quoted 18 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if
the program istail_call_reachable, regardless of whether the function is a
subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore,
it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)Quoted 17 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2130
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { + bool pseudo_call = src_reg == BPF_PSEUDO_CALL; + bool subprog_tail_call_reachable = dst_reg; u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + if (pseudo_call && subprog_tail_call_reachable) {Why we need
subprog_tail_call_reachable? Does
tail_call_reachable &&psueudo_call
work the same way?
Quoted 7 lines
'
tail_call_reachable&&pseudo_call' works too. However, it will propagate tailcall info to subprog even if the subprog is nottail_call_reachable.
subprog_tail_call_reachableindicates the subprog requires tailcall info from its caller.
So, 'pseudo_call&&subprog_tail_call_reachable' is better.
In verifier.c, we have
func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable;
that is subprog_info tail_call_reachable has been transferred to func[i] tail_call_reachable.
In x86 do_jit() func, we have
bool tail_call_reachable = bpf_prog->aux->tail_call_reachable
So looks like we do not need verifier.c change here.
Did I miss anything? Could you give a concrete example to show subprog_tail_call_reachable approach is better than tail_call_reachable?
Quoted 4 lines
Thanks,
Leon
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 54 lines · Yonghong Song
Quoted 38 lines · Leon Hwang
Quoted 19 lines · Yonghong Song
Quoted 18 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if
the program istail_call_reachable, regardless of whether the function is a
subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore,
it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>---
arch/x86/net/bpf_jit_comp.c| 4 +++-
kernel/bpf/verifier.c | 6 ++++++2 files changed, 9 insertions(+), 1 deletion(-)Quoted 17 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2130
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { + bool pseudo_call = src_reg == BPF_PSEUDO_CALL; + bool subprog_tail_call_reachable = dst_reg; u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + if (pseudo_call && subprog_tail_call_reachable) {Why we need
subprog_tail_call_reachable? Does
tail_call_reachable &&psueudo_call
work the same way?Quoted 7 lines
'
tail_call_reachable&&pseudo_call' works too. However, it will propagate tailcall info to subprog even if the subprog is nottail_call_reachable.
subprog_tail_call_reachableindicates the subprog requires tailcall info from its caller.
So, 'pseudo_call&&subprog_tail_call_reachable' is better.In
verifier.c, we have
func[i]->aux->tail_call_reachable = env-Quoted 1 line
subprog_info[i].tail_call_reachable;
that is
subprog_infotail_call_reachablehas been transferred to func[i]tail_call_reachable.
Quoted 7 lines
In x86
do_jit()func, we have
booltail_call_reachable=bpf_prog->aux->tail_call_reachableSo looks like we do not need
verifier.cchange here.
Did I miss anything? Could you give a concrete example to showsubprog_tail_call_reachableapproach is better thantail_call_reachable?
Sure, here's an example:
struct {
__uint(type, BPF_MAP_TYPE_PROG_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
__uint(max_entries, 1);
} jmp_table SEC(".maps");static __noinline intsubprog_tc1(struct __sk_buff *skb)
{
volatile int retval = TC_ACT_OK;
bpf_tail_call_static(skb, jmp_table, 0);
return retval;
}static __noinline intsubprog_tc2(struct __sk_buff *skb)
{
volatile int retval = TC_ACT_OK;
return retval;
}SEC("tc")
int entry_tc(struct __sk_buff *skb)
{
u32 pid = bpf_get_smp_processor_id();
// do something with pid
subprog_tc2(skb);
return subprog_tc1(skb);
}From the verifier's perspective, both entry_tc and subprog_tc1 are tail_call_reachable.
When handling 'BPF_JMP | BPF_CALL' in the x86 do_jit() for entry_tc, three cases arise:
1. bpf_get_smp_processor_id()
2. subprog_tc1()
3. subprog_tc2()
At this point in x86 do_jit() for entry_tc, entry_tc is considered tail_call_reachable. The check 'bool pseudo_call = src_reg == BPF_PSEUDO_CALL' is used to determine whether to call a subprogram.
The question is: when should tailcall info be propagated? Should it be when entry_tc is tail_call_reachable, even if subprog_tc2 is called, or when subprog_tc1 is specifically tail_call_reachable?
I believe it is better to propagate the tailcall info when subprog_tc1 is tail_call_reachable.
Thanks,
Leon
Yonghong Song
reviewerRe: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 109 lines · Leon Hwang
Quoted 51 lines · Yonghong Song
Quoted 37 lines · Leon Hwang
Quoted 19 lines · Yonghong Song
Quoted 18 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if
the program istail_call_reachable, regardless of whether the function is a
subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore,
it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)Quoted 17 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2130
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { + bool pseudo_call = src_reg == BPF_PSEUDO_CALL; + bool subprog_tail_call_reachable = dst_reg; u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + if (pseudo_call && subprog_tail_call_reachable) {Why we need
subprog_tail_call_reachable? Does
tail_call_reachable &&psueudo_call
work the same way?Quoted 7 lines
'
tail_call_reachable&&pseudo_call' works too. However, it will propagate tailcall info to subprog even if the subprog is nottail_call_reachable.
subprog_tail_call_reachableindicates the subprog requires tailcall info from its caller.
So, 'pseudo_call&&subprog_tail_call_reachable' is better.In
verifier.c, we have
func[i]->aux->tail_call_reachable = env-Quoted 1 line
subprog_info[i].tail_call_reachable;
that is
subprog_infotail_call_reachablehas been transferred to func[i]tail_call_reachable.Quoted 7 lines
In x86
do_jit()func, we have
booltail_call_reachable=bpf_prog->aux->tail_call_reachableSo looks like we do not need
verifier.cchange here.
Did I miss anything? Could you give a concrete example to showsubprog_tail_call_reachableapproach is better thantail_call_reachable?Sure, here's an example:
struct { __uint(type, BPF_MAP_TYPE_PROG_ARRAY); __uint(key_size, sizeof(u32)); __uint(value_size, sizeof(u32)); __uint(max_entries, 1); } jmp_table SEC(".maps");static
__noinlineintsubprog_tc1(struct __sk_buff *skb){ volatile int retval = TC_ACT_OK; bpf_tail_call_static(skb, jmp_table, 0); return retval; }static
__noinlineintsubprog_tc2(struct __sk_buff *skb){ volatile int retval = TC_ACT_OK; return retval; }
SEC("tc")
intentry_tc(struct __sk_buff *skb){ u32 pid = bpf_get_smp_processor_id(); // do something with pid subprog_tc2(skb); return subprog_tc1(skb); }From the verifier's perspective, both
entry_tcandsubprog_tc1aretail_call_reachable.When handling '
BPF_JMP|BPF_CALL' in the x86do_jit()forentry_tc, three cases arise:1.
bpf_get_smp_processor_id()
2.subprog_tc1()
3.subprog_tc2()At this point in x86
do_jit()forentry_tc,entry_tcis consideredtail_call_reachable. The check 'boolpseudo_call=src_reg==BPF_PSEUDO_CALL' is used to determine whether to call a subprogram.
Quoted 6 lines
The question is: when should tailcall info be propagated? Should it be when
entry_tcistail_call_reachable, even ifsubprog_tc2is called, or whensubprog_tc1is specificallytail_call_reachable?I believe it is better to propagate the tailcall info when
subprog_tc1istail_call_reachable.
Okay, I see. Thanks for explanation.
You use the insn->dst_reg to record whether callee is tail call reachable or not. I think you can reuse insn->off which currently represents subprog number but it is not used for jit. We can use that to indicate callee is tail call reachable or not.
Something like below:
arch/x86/net/bpf_jit_comp.c +2 -1
@@ -2127,7 +2127,8 @@ st: if (is_imm8(insn->off))u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +/* insn->off == 1 means the callee is tail call reachable */ +if (src_reg ==BPF_PSEUDO_CALL && insn->off ==1 ) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }
kernel/bpf/verifier.c +2 -0
@@ -20096,6 +20096,8 @@ static int jit_subprogs(struct bpf_verifier_env *env)continue ; subprog = insn->off; insn->imm =BPF_CALL_IMM (func[subprog]->bpf_func); +/* Indicate whether callee is tail call reachable or not */ + insn->off = func[subprog]->aux->tail_call_reachable; }
WDYT?
Quoted 4 lines
Thanks,
Leon
Yonghong Song
reviewerRe: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 157 lines · Yonghong Song
Quoted 108 lines · Leon Hwang
Quoted 51 lines · Yonghong Song
Quoted 37 lines · Leon Hwang
Quoted 19 lines · Yonghong Song
Quoted 18 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if
the program istail_call_reachable, regardless of whether the function is a
subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore,
it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>---
arch/x86/net/bpf_jit_comp.c| 4 +++-
kernel/bpf/verifier.c | 6 ++++++2 files changed, 9 insertions(+), 1 deletion(-)Quoted 17 lines
arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2130
Quoted 13 lines
arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off)) /* call */ case BPF_JMP | BPF_CALL: { + bool pseudo_call = src_reg == BPF_PSEUDO_CALL; + bool subprog_tail_call_reachable = dst_reg; u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + if (pseudo_call && subprog_tail_call_reachable) {Why we need
subprog_tail_call_reachable? Does
tail_call_reachable &&psueudo_call
work the same way?Quoted 7 lines
'
tail_call_reachable&&pseudo_call' works too. However, it will propagate tailcall info to subprog even if the subprog is nottail_call_reachable.
subprog_tail_call_reachableindicates the subprog requires tailcall info from its caller.
So, 'pseudo_call&&subprog_tail_call_reachable' is better.In
verifier.c, we have
func[i]->aux->tail_call_reachable = env-Quoted 1 line
subprog_info[i].tail_call_reachable;
that is
subprog_infotail_call_reachablehas been transferred to func[i]tail_call_reachable.Quoted 6 lines
In x86
do_jit()func, we have
booltail_call_reachable=bpf_prog->aux->tail_call_reachableSo looks like we do not need
verifier.cchange here.
Did I miss anything? Could you give a concrete example to showsubprog_tail_call_reachableapproach is better thantail_call_reachable?Sure, here's an example:
struct {
__uint(type,BPF_MAP_TYPE_PROG_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
__uint(max_entries, 1);
}jmp_tableSEC(".maps");static
__noinlineintsubprog_tc1(struct __sk_buff *skb)
{
volatile int retval =TC_ACT_OK;bpf_tail_call_static(skb,
jmp_table, 0);
return retval;
}static
__noinlineintsubprog_tc2(struct __sk_buff *skb)
{
volatile int retval =TC_ACT_OK;return retval;
}
SEC("tc")
intentry_tc(struct __sk_buff *skb)
{
u32 pid =bpf_get_smp_processor_id();
// do something with pid
subprog_tc2(skb);
returnsubprog_tc1(skb);
}From the verifier's perspective, both
entry_tcandsubprog_tc1aretail_call_reachable.When handling '
BPF_JMP|BPF_CALL' in the x86do_jit()forentry_tc, three cases arise:1.
bpf_get_smp_processor_id()
2.subprog_tc1()
3.subprog_tc2()At this point in x86
do_jit()forentry_tc,entry_tcis consideredtail_call_reachable. The check 'boolpseudo_call=src_reg==BPF_PSEUDO_CALL' is used to determine whether to call a subprogram.Quoted 6 lines
The question is: when should tailcall info be propagated? Should it be when
entry_tcistail_call_reachable, even ifsubprog_tc2is called, or whensubprog_tc1is specificallytail_call_reachable?I believe it is better to propagate the tailcall info when
subprog_tc1istail_call_reachable.Okay, I see. Thanks for explanation.
You use the
insn->dst_regto record whether callee is tail call reachable or not. I think you can reuseinsn->offwhich currently represents subprog number but it is not used for jit. We can use that to indicate callee is tail call reachable or not.Something like below:
arch/x86/net/bpf_jit_comp.c +1
-1@@ -2127,7 +2127,8 @@ st: if (is_imm8(insn->off)) u8 *ip = image + addrs[i - 1]; func = (u8 *) __bpf_call_base + imm32; - if (tail_call_reachable) { + /* insn->off == 1 means the callee is tailcall reachable */
+ if (src_reg==BPF_PSEUDO_CALL&&insn->off==
1) {LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
ip += 7;
}kernel/bpf/verifier.c +1
-0@@ -20096,6 +20096,8 @@ static int jit_subprogs(struct bpf_verifier_env *env) continue; subprog = insn->off; insn->imm = BPF_CALL_IMM(func[subprog]->bpf_func); + /* Indicate whether callee is tail callreachable or not */
+insn->off=
func[subprog]->aux->tail_call_reachable;
}
Quoted 1 line
WDYT?
Sorry, the above seems not working since verifier do jit twice for the same prog and two jit'ed results need to be the same. The above change could make jit result different between two passes.
Quoted 6 lines
Quoted 5 lines
Thanks,
Leon
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
In reply to Leon HwangQuoted 10 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if the program istail_call_reachable, regardless of whether the function is a subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore, it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>
LGTM with a nit below.
Quoted 28 lines
--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ; +bool subprog_tail_call_reachable = dst_reg;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && subprog_tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static int jit_subprogs(struct bpf_verifier_env *env) · line 19998
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -19990,6 +19990,12 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn[0 ].imm = (u32 )addr; insn[1 ].imm = addr >>32 ; } + +if (bpf_pseudo_call (insn)) +/* In the x86_64 JIT, tailcall information can only be +* propagated if the subprog is tail_call_reachable. +*/ + insn->dst_reg = env->subprog_info[subprog].tail_call_reachable;
The comment can be simplied as
/* Indicate whether callee is tail call
reachable or not */
Quoted 3 lines
kernel/bpf/verifier.c +0
-0} err =bpf_prog_alloc_jited_linfo (prog);
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
In reply to Leon HwangQuoted 39 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if the program istail_call_reachable, regardless of whether the function is a subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore, it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ; +bool subprog_tail_call_reachable = dst_reg;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && subprog_tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static int jit_subprogs(struct bpf_verifier_env *env) · line 19998
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -19990,6 +19990,12 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn[0 ].imm = (u32 )addr; insn[1 ].imm = addr >>32 ; } + +if (bpf_pseudo_call (insn)) +/* In the x86_64 JIT, tailcall information can only be +* propagated if the subprog is tail_call_reachable. +*/ + insn->dst_reg = env->subprog_info[subprog].tail_call_reachable;
I really don't like hacking flags into dst_reg.
We already abuse insn->off which is ugly too,
but at least we clean insns later after JIT.
I'd rather live with this tail call inefficiency than abuse insns fields further.
pw-bot: cr
Re: [PATCH bpf-next 1/2] bpf, x64: Propagate tailcall info only for tail_call_reachable subprogs
Quoted 54 lines · Alexei Starovoitov
Quoted 39 lines · Leon Hwang
In the
x86_64JIT, when calling a function, tailcall info is propagated if the program istail_call_reachable, regardless of whether the function is a subprog, helper, or kfunc. However, this propagation is unnecessary for not-tail_call_reachable subprogs, helpers, or kfuncs.The verifier can determine if a subprog is
tail_call_reachable. Therefore, it can be optimized to only propagate tailcall info when the callee is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>--- arch/x86/net/bpf_jit_comp.c | 4 +++- kernel/bpf/verifier.c | 6 ++++++ 2 files changed, 9 insertions(+), 1 deletion(-)arch/x86/net/bpf_jit_comp.c +3
-1@@ -2124,10 +2124,12 @@ st: if (is_imm8(insn->off))/* call */ case BPF_JMP |BPF_CALL : { +bool pseudo_call = src_reg ==BPF_PSEUDO_CALL ; +bool subprog_tail_call_reachable = dst_reg;u8 *ip = image + addrs[i -1 ]; func = (u8 *) __bpf_call_base + imm32; -if (tail_call_reachable) { +if (pseudo_call && subprog_tail_call_reachable) {LOAD_TAIL_CALL_CNT_PTR (bpf_prog->aux->stack_depth); ip +=7 ; }kernel/bpf/verifier.c +0
-0kernel/bpf/verifier.c static int jit_subprogs(struct bpf_verifier_env *env) · line 19998
Quoted 9 lines
kernel/bpf/verifier.c +6
-0@@ -19990,6 +19990,12 @@ static int jit_subprogs(struct bpf_verifier_env *env) insn[0 ].imm = (u32 )addr; insn[1 ].imm = addr >>32 ; } + +if (bpf_pseudo_call (insn)) +/* In the x86_64 JIT, tailcall information can only be +* propagated if the subprog is tail_call_reachable. +*/ + insn->dst_reg = env->subprog_info[subprog].tail_call_reachable;I really don't like hacking flags into
dst_reg.
We already abuseinsn->offwhich is ugly too,
but at least we clean insns later after JIT.
Quoted 3 lines
I'd rather live with this tail call inefficiency than abuse insns fields further.
OK, let us use 'pseudo_call && tail_call_reachable' in x86 JIT to avoid touching 'insn->dst_reg'.
Thanks,
Leon
Quoted 1 line
pw-bot: cr
PATCH 2/2 bpf, verifier: Check trampoline target is tail_call_reachable subprog
In reply to Leon HwangIn the x86_64 JIT, tailcall info is propagated through the trampoline when the target program is tail_call_reachable. However, this propagation is unnecessary if the target is a main prog, or a subprog that is not tail_call_reachable.
Since the verifier can determine if a subprog is tail_call_reachable, it should only propagate tailcall info when the target is subprog and the subprog is actually tail_call_reachable.
--- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 4 +++- 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/bpf.h +1 -0
@@ -1220,6 +1220,7 @@ struct bpf_attach_target_info {struct module *tgt_mod;const char *tgt_name;const struct btf_type *tgt_type; +bool tgt_tail_call_reachable; };#define BPF_DISPATCHER_MAX 48/* Fits in 2048B */
kernel/bpf/verifier.c +3 -1
@@ -21916,6 +21916,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,bpf_log (log,"Subprog %s doesn't exist\n" , tname);return -EINVAL ; } + tgt_info->tgt_tail_call_reachable = subprog && + aux->func[subprog]->aux->tail_call_reachable;if (aux->func && aux->func[subprog]->aux->exception_cb) { bpf_log(log, "%s programs cannot attach to exception callback\n",
@@ -22285,7 +22287,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)if (!tr)return -ENOMEM ; -if (tgt_prog && tgt_prog->aux->tail_call_reachable) +if (tgt_prog && tgt_info.tgt_tail_call_reachable) tr->flags =BPF_TRAMP_F_TAIL_CALL_CTX ; prog->aux->dst_trampoline = tr;
Signature · 1 line
2.44.0
Re: [PATCH bpf-next 2/2] bpf, verifier: Check trampoline target is tail_call_reachable subprog
Quoted 10 lines · Leon Hwang
In the
x86_64JIT, tailcall info is propagated through the trampoline when the target program istail_call_reachable. However, this propagation is unnecessary if the target is a main prog, or a subprog that is nottail_call_reachable.Since the verifier can determine if a subprog is
tail_call_reachable, it should only propagate tailcall info when the target is subprog and the subprog is actuallytail_call_reachable.Signed-off-by Leon Hwang <[email protected]>
Looks correct to me.