Skip to contentSkip to conversation

bpf, x64: Introduce two tailcall enhancements

PATCH In review Leon Hwang 4 participants 8 messages Nov 7, 2024 – Nov 13, 2024

Subsystems (4)
lore
Share & tools
lore

Leon Hwang

author
#0 lore raw

v3 cover bpf, x64: Introduce two tailcall enhancements

This patch set introduces two enhancements aimed at improving tailcall handling in the x64 JIT:

1. Tailcall info is propagated only for subprogs.
2. Tailcall info is propagated through the trampoline only when the target is a subprog and it is tail_call_reachable.

Changes:
v2 -> v3:
* Add Yonghong's ACK.

v1 -> v2:
* Address comment from Alexei:
* Rather live with tail call inefficiency than abuse insns fields further.

Leon Hwang (2):
bpf, x64: Propagate tailcall info only for subprogs
bpf, verifier: Check trampoline target is tail_call_reachable subprog

 arch/x86/net/bpf_jit_comp.c | 3 ++-
 include/linux/bpf.h         | 1 +
 kernel/bpf/verifier.c       | 4 +++-
 3 files changed, 6 insertions(+), 2 deletions(-)
Signature · 1 line
2.44.0
author
#1 lore raw

PATCH v3 1/2 bpf, x64: Propagate tailcall info only for subprogs

In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.

---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +2 -1
index 06b080b61aa57..eb08cc6d66401 100644
@@ -2124,10 +2124,11 @@ st:			if (is_imm8(insn->off))
 
 			/* call */
 		case BPF_JMP | BPF_CALL: {
+			bool pseudo_call = src_reg == BPF_PSEUDO_CALL;
 			u8 *ip = image + addrs[i - 1];
 
 			func = (u8 *) __bpf_call_base + imm32;
-			if (tail_call_reachable) {
+			if (pseudo_call && tail_call_reachable) {
 				LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
 				ip += 7;
 			}
Signature · 1 line
2.44.0
#2 lore raw

Re: [PATCH bpf-next v3 1/2] bpf, x64: Propagate tailcall info only for subprogs

Quoted 17 lines · Leon Hwang

In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.

---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +0 -0
index 06b080b61aa57..eb08cc6d66401 100644
@@ -2124,10 +2124,11 @@ st:                     if (is_imm8(insn->off))
 
                        /* call */

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2134

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2 -1
                case BPF_JMP | BPF_CALL: {
+                       bool pseudo_call = src_reg == BPF_PSEUDO_CALL;
                        u8 *ip = image + addrs[i - 1];
 
                        func = (u8 *) __bpf_call_base + imm32;
-                       if (tail_call_reachable) {
+                       if (pseudo_call && tail_call_reachable) {
                                LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
                                ip += 7;
                        }

I've applied this patch with this tweak:
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable)

I don't see much value in patch 2.
The tail_call feature is an old approach. It is now causing maintenance issues with other features.
I'd rather not touch anything tail call related. So I dropped patch 2.

I'd like to see proper indirect goto and indirect call support being developed further.
Anton started working on it, but dropped the ball. We need to commandeer the patches.

author
#3 lore raw

Re: [PATCH bpf-next v3 1/2] bpf, x64: Propagate tailcall info only for subprogs

Quoted 38 lines · Alexei Starovoitov
Quoted 17 lines · Leon Hwang

In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.

---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +0 -0
index 06b080b61aa57..eb08cc6d66401 100644
@@ -2124,10 +2124,11 @@ st:                     if (is_imm8(insn->off))
 
                        /* call */

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2134

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2 -1
                case BPF_JMP | BPF_CALL: {
+                       bool pseudo_call = src_reg == BPF_PSEUDO_CALL;
                        u8 *ip = image + addrs[i - 1];
 
                        func = (u8 *) __bpf_call_base + imm32;
-                       if (tail_call_reachable) {
+                       if (pseudo_call && tail_call_reachable) {
                                LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
                                ip += 7;
                        }

I've applied this patch with this tweak:
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable)

I don't see much value in patch 2.
The tail_call feature is an old approach. It is now causing maintenance issues with other features.
I'd rather not touch anything tail call related. So I dropped patch 2.

Quoted 4 lines

I'd like to see proper indirect goto and indirect call support being developed further.
Anton started working on it, but dropped the ball. We need to commandeer the patches.

Great to see jmp table supporting tail call.

Thanks,
Leon

#4 lore raw

Re: [PATCH bpf-next v3 1/2] bpf, x64: Propagate tailcall info only for subprogs

In reply to Alexei Starovoitov
Quoted 38 lines · Alexei Starovoitov
Quoted 17 lines · Leon Hwang

In x64 JIT, propagate tailcall info only for subprogs, not for helpers or kfuncs.

---
 arch/x86/net/bpf_jit_comp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/net/bpf_jit_comp.c +0 -0
index 06b080b61aa57..eb08cc6d66401 100644
@@ -2124,10 +2124,11 @@ st:                     if (is_imm8(insn->off))
 
                        /* call */

arch/x86/net/bpf_jit_comp.c st: if (is_imm8(insn->off)) · line 2134

Quoted 10 lines
arch/x86/net/bpf_jit_comp.c +2 -1
                case BPF_JMP | BPF_CALL: {
+                       bool pseudo_call = src_reg == BPF_PSEUDO_CALL;
                        u8 *ip = image + addrs[i - 1];
 
                        func = (u8 *) __bpf_call_base + imm32;
-                       if (tail_call_reachable) {
+                       if (pseudo_call && tail_call_reachable) {
                                LOAD_TAIL_CALL_CNT_PTR(bpf_prog->aux->stack_depth);
                                ip += 7;
                        }

I've applied this patch with this tweak:
if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable)

I don't see much value in patch 2.
The tail_call feature is an old approach. It is now causing maintenance issues with other features.
I'd rather not touch anything tail call related. So I dropped patch 2.

Quoted 4 lines

I'd like to see proper indirect goto and indirect call support being developed further.
Anton started working on it, but dropped the ball. We need to commandeer the patches.

Actually, I am still on it. I will start sending patches this week.

#5 lore raw

Re: [PATCH bpf-next v3 1/2] bpf, x64: Propagate tailcall info only for subprogs

Quoted 2 lines · Anton Protopopov

Actually, I am still on it. I will start sending patches this week.

Ohh. Great. Cannot wait.

author
#6 lore raw

PATCH v3 2/2 bpf, verifier: Check trampoline target is tail_call_reachable subprog

In reply to Leon Hwang

In the x86_64 JIT, tailcall info is propagated through the trampoline when the target program is tail_call_reachable. However, this propagation is unnecessary if the target is a main prog, or a subprog that is not tail_call_reachable.

Since the verifier can determine if a subprog is tail_call_reachable, it should only propagate tailcall info when the target is subprog and the subprog is actually tail_call_reachable.

---
 include/linux/bpf.h   | 1 +
 kernel/bpf/verifier.c | 4 +++-
 2 files changed, 4 insertions(+), 1 deletion(-)
include/linux/bpf.h +1 -0
index 1b84613b10ac7..1e5b439ab4c7d 100644
@@ -1253,6 +1253,7 @@ struct bpf_attach_target_info {
 	struct module *tgt_mod;
 	const char *tgt_name;
 	const struct btf_type *tgt_type;
+	bool tgt_tail_call_reachable;
 };
 
 #define BPF_DISPATCHER_MAX 48 /* Fits in 2048B */
kernel/bpf/verifier.c +3 -1
index 7958d6ff6b739..e5b9754a2f0be 100644
@@ -22004,6 +22004,8 @@ int bpf_check_attach_target(struct bpf_verifier_log *log,
 			bpf_log(log, "Subprog %s doesn't exist\n", tname);
 			return -EINVAL;
 		}
+		tgt_info->tgt_tail_call_reachable = subprog &&
+						    aux->func[subprog]->aux->tail_call_reachable;
 		if (aux->func && aux->func[subprog]->aux->exception_cb) {
 			bpf_log(log,
 				"%s programs cannot attach to exception callback\n",
@@ -22373,7 +22375,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
 	if (!tr)
 		return -ENOMEM;
 
-	if (tgt_prog && tgt_prog->aux->tail_call_reachable)
+	if (tgt_prog && tgt_info.tgt_tail_call_reachable)
 		tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX;
 
 	prog->aux->dst_trampoline = tr;
Signature · 1 line
2.44.0
#7 lore raw
In reply to Leon Hwang
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 8 lines · On Thu, 7 Nov 2024 21:45:27 +0800 you wrote:

This patch set introduces two enhancements aimed at improving tailcall handling in the x64 JIT:

1. Tailcall info is propagated only for subprogs.
2. Tailcall info is propagated through the trampoline only when the target is a subprog and it is tail_call_reachable.

[...]

Here is the summary with links:
- [bpf-next,v3,1/2] bpf, x64: Propagate tailcall info only for subprogs

    https://git.kernel.org/bpf/bpf-next/c/a1087da9d11e

- [bpf-next,v3,2/2] bpf, verifier: Check trampoline target is tail_call_reachable subprog (no matching commit)

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html