Skip to contentSkip to conversation

LoongArch: Fix BPF trampoline related issues

PATCH Superseded Hengqi Chen 2 participants 19 messages Sep 3, 2025 – Sep 4, 2025

Subsystems (3)

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Hengqi Chen

author reviewer
#0 lore raw

v4 cover LoongArch: Fix BPF trampoline related issues

The following two selftest cases triggers oops on LoongArch:

    $ ./test_progs -a ns_bpf_qdisc -a tracing_struct

This series tries to fix/workaround these issues.
See individual commit for details.

v3 -> v4:
* Completely rework of previous version
* Fix more subtle issues

Hengqi Chen (8):
LoongArch: BPF: Remove duplicated flags check
LoongArch: BPF: Remove duplicated bpf_flush_icache()
LoongArch: BPF: No support of struct argument in trampoline programs
LoongArch: BPF: No text_poke() for kernel text
LoongArch: BPF: Don't assume trampoline size is page aligned
LoongArch: BPF: Make trampoline size stable
LoongArch: BPF: Make error handling robust in
arch_prepare_bpf_trampoline()
LoongArch: BPF: Sign extend struct ops return values properly

 arch/loongarch/net/bpf_jit.c | 67 +++++++++++++++++++++++++++---------
 1 file changed, 51 insertions(+), 16 deletions(-)

--
2.43.5

author reviewer
#1 lore raw

PATCH v4 1/8 LoongArch: BPF: Remove duplicated flags check

The check for (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY) is duplicated in __arch_prepare_bpf_trampoline(). Remove it.

While at it, make sure stack_size and nargs are initialized once.

---
 arch/loongarch/net/bpf_jit.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)
arch/loongarch/net/bpf_jit.c +2 -7
index abfdb6bb5c38..77033947f1b2 100644
@@ -1453,7 +1453,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 					 void *func_addr, u32 flags)
 {
 	int i, ret, save_ret;
-	int stack_size = 0, nargs = 0;
+	int stack_size, nargs;
 	int retval_off, args_off, nargs_off, ip_off, run_ctx_off, sreg_off, tcc_ptr_off;
 	bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT;
 	void *orig_call = func_addr;
@@ -1462,9 +1462,6 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 	struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
 	u32 **branches = NULL;
 
-	if (flags & (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY))
-		return -ENOTSUPP;
-
 	/*
 	 * FP + 8       [ RA to parent func ] return address to parent
 	 *                    function
@@ -1498,10 +1495,8 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 	if (flags & (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY))
 		return -ENOTSUPP;
 
-	stack_size = 0;
-
 	/* Room of trampoline frame to store return address and frame pointer */
-	stack_size += 16;
+	stack_size = 16;
 
 	save_ret = flags & (BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_RET_FENTRY_RET);
 	if (save_ret) {
Signature · 1 line
2.43.5
author reviewer
#2 lore raw

PATCH v4 2/8 LoongArch: BPF: Remove duplicated bpf_flush_icache()

In reply to Hengqi Chen

The bpf_flush_icache() is called by bpf_arch_text_copy() already. So remove it.

---
 arch/loongarch/net/bpf_jit.c | 1 -
 1 file changed, 1 deletion(-)
arch/loongarch/net/bpf_jit.c +0 -1
index 77033947f1b2..9155f9e725a1 100644
@@ -1721,7 +1721,6 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
 		goto out;
 	}
 
-	bpf_flush_icache(ro_image, ro_image_end);
 out:
 	kvfree(image);
 	return ret < 0 ? ret : size;
Signature · 1 line
2.43.5
maintainer
#3 lore raw

Re: [PATCH v4 2/8] LoongArch: BPF: Remove duplicated bpf_flush_icache()

Hi, Hengqi,

Quoted 14 lines · Hengqi Chen

The bpf_flush_icache() is called by bpf_arch_text_copy() already. So remove it.

---
 arch/loongarch/net/bpf_jit.c | 1 -
 1 file changed, 1 deletion(-)
arch/loongarch/net/bpf_jit.c +0 -0
index 77033947f1b2..9155f9e725a1 100644

arch/loongarch/net/bpf_jit.c int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · line 1724

Quoted 4 lines
arch/loongarch/net/bpf_jit.c +0 -1
@@ -1721,7 +1721,6 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
                goto out;
        }
 
-       bpf_flush_icache(ro_image, ro_image_end);

Both ARM64 and RISC-V do this, so I prefer to keep it.

Huacai

Quoted 6 lines
arch/loongarch/net/bpf_jit.c +0 -1
 out:
        kvfree(image);
        return ret < 0 ? ret : size;
--

2.43.5

author reviewer
#4 lore raw

Re: [PATCH v4 2/8] LoongArch: BPF: Remove duplicated bpf_flush_icache()

Quoted 21 lines · Huacai Chen

Hi, Hengqi,

Quoted 17 lines · Hengqi Chen

The bpf_flush_icache() is called by bpf_arch_text_copy() already. So remove it.

---
 arch/loongarch/net/bpf_jit.c | 1 -
 1 file changed, 1 deletion(-)
arch/loongarch/net/bpf_jit.c +0 -0
index 77033947f1b2..9155f9e725a1 100644
@@ -1721,7 +1721,6 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
                goto out;
        }
 
Quoted 3 lines

arch/loongarch/net/bpf_jit.c int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · line 1724

Quoted 1 line
arch/loongarch/net/bpf_jit.c +0 -1
-       bpf_flush_icache(ro_image, ro_image_end);

Both ARM64 and RISC-V do this, so I prefer to keep it.

Then they probably should remove it too, I guess.

Quoted 8 lines

Huacai

Quoted 6 lines
arch/loongarch/net/bpf_jit.c +0 -1
 out:
        kvfree(image);
        return ret < 0 ? ret : size;
--

2.43.5

maintainer
#5 lore raw

Re: [PATCH v4 2/8] LoongArch: BPF: Remove duplicated bpf_flush_icache()

Quoted 23 lines · Hengqi Chen
Quoted 21 lines · Huacai Chen

Hi, Hengqi,

Quoted 17 lines · Hengqi Chen

The bpf_flush_icache() is called by bpf_arch_text_copy() already. So remove it.

---
 arch/loongarch/net/bpf_jit.c | 1 -
 1 file changed, 1 deletion(-)
arch/loongarch/net/bpf_jit.c +0 -0
index 77033947f1b2..9155f9e725a1 100644
@@ -1721,7 +1721,6 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
                goto out;
        }
 
Quoted 5 lines
Quoted 3 lines

arch/loongarch/net/bpf_jit.c int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · line 1724

Quoted 1 line
arch/loongarch/net/bpf_jit.c +0 -1
-       bpf_flush_icache(ro_image, ro_image_end);

Both ARM64 and RISC-V do this, so I prefer to keep it.

Then they probably should remove it too, I guess.

I think we should understand why they do so first.

Huacai

Quoted 9 lines
Quoted 9 lines

Huacai

Quoted 6 lines
arch/loongarch/net/bpf_jit.c +0 -1
 out:
        kvfree(image);
        return ret < 0 ? ret : size;
--

2.43.5

author reviewer
#6 lore raw

Re: [PATCH v4 2/8] LoongArch: BPF: Remove duplicated bpf_flush_icache()

Quoted 25 lines · Huacai Chen
Quoted 23 lines · Hengqi Chen
Quoted 21 lines · Huacai Chen

Hi, Hengqi,

Quoted 17 lines · Hengqi Chen

The bpf_flush_icache() is called by bpf_arch_text_copy() already. So remove it.

---
 arch/loongarch/net/bpf_jit.c | 1 -
 1 file changed, 1 deletion(-)
arch/loongarch/net/bpf_jit.c +0 -0
index 77033947f1b2..9155f9e725a1 100644
@@ -1721,7 +1721,6 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
                goto out;
        }
 
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

arch/loongarch/net/bpf_jit.c int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · line 1724

Quoted 1 line
arch/loongarch/net/bpf_jit.c +0 -1
-       bpf_flush_icache(ro_image, ro_image_end);

Both ARM64 and RISC-V do this, so I prefer to keep it.

Then they probably should remove it too, I guess.

I think we should understand why they do so first.

Just got confirmation (offlist) from Huawei folk, we could remove it.

Quoted 11 lines

Huacai

Quoted 9 lines
Quoted 9 lines

Huacai

Quoted 6 lines
arch/loongarch/net/bpf_jit.c +0 -1
 out:
        kvfree(image);
        return ret < 0 ? ret : size;
--

2.43.5

author reviewer
#7 lore raw

PATCH v4 3/8 LoongArch: BPF: No support of struct argument in trampoline programs

In reply to Hengqi Chen

The current implementation does not support struct argument. This cause a oops when running bpf selftest:

    $ ./test_progs -a tracing_struct
    CPU -1 Unable to handle kernel paging request at virtual address 0000000000000018, era == 90000000845659f4, ra == 90000000845659e8
    Oops[#1]:
    CPU -1 Unable to handle kernel paging request at virtual address 0000000000000018, era == 9000000085bef268, ra == 90000000844f3938
    rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
    rcu:     1-...0: (19 ticks this GP) idle=1094/1/0x4000000000000000 softirq=1380/1382 fqs=801
    rcu:     (detected by 0, t=5252 jiffies, g=1197, q=52 ncpus=4)
    Sending NMI from CPU 0 to CPUs 1:
    rcu: rcu_preempt kthread starved for 2495 jiffies! g1197 f0x0 RCU_GP_DOING_FQS(6) ->state=0x0 ->cpu=2
    rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
    rcu: RCU grace-period kthread stack dump:
    task:rcu_preempt     state:I stack:0     pid:15    tgid:15    ppid:2      task_flags:0x208040 flags:0x00000800
    Stack : 9000000100423e80 0000000000000402 0000000000000010 90000001003b0680
            9000000085d88000 0000000000000000 0000000000000040 9000000087159350
            9000000085c2b9b0 0000000000000001 900000008704a000 0000000000000005
            00000000ffff355b 00000000ffff355b 0000000000000000 0000000000000004
            9000000085d90510 0000000000000000 0000000000000002 7b5d998f8281e86e
            00000000ffff355c 7b5d998f8281e86e 000000000000003f 9000000087159350
            900000008715bf98 0000000000000005 9000000087036000 900000008704a000
            9000000100407c98 90000001003aff80 900000008715c4c0 9000000085c2b9b0
            00000000ffff355b 9000000085c33d3c 00000000000000b4 0000000000000000
            9000000007002150 00000000ffff355b 9000000084615480 0000000007000002
            ...
    Call Trace:
    [<9000000085c2a868>] __schedule+0x410/0x1520
    [<9000000085c2b9ac>] schedule+0x34/0x190
    [<9000000085c33d38>] schedule_timeout+0x98/0x140
    [<90000000845e9120>] rcu_gp_fqs_loop+0x5f8/0x868
    [<90000000845ed538>] rcu_gp_kthread+0x260/0x2e0
    [<900000008454e8a4>] kthread+0x144/0x238
    [<9000000085c26b60>] ret_from_kernel_thread+0x28/0xc8
    [<90000000844f20e4>] ret_from_kernel_thread_asm+0xc/0x88

rcu: Stack dump where RCU GP kthread last ran:
Sending NMI from CPU 0 to CPUs 2:
NMI backtrace for cpu 2 skipped: idling at idle_exit+0x0/0x4

Reject it for now.

Fixes f9b6b41f0cf3 ("LoongArch: BPF: Add basic bpf trampoline support")
---
 arch/loongarch/net/bpf_jit.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/loongarch/net/bpf_jit.c +6 -0
index 9155f9e725a1..7b7e449b9ea9 100644
@@ -1492,6 +1492,12 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 	if (m->nr_args > LOONGARCH_MAX_REG_ARGS)
 		return -ENOTSUPP;
 
+	/* don't support struct argument */
+	for (i = 0; i < m->nr_args; i++) {
+		if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)
+			return -ENOTSUPP;
+	}
+
 	if (flags & (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY))
 		return -ENOTSUPP;
 
Signature · 1 line
2.43.5
author reviewer
#8 lore raw

PATCH v4 4/8 LoongArch: BPF: No text_poke() for kernel text

In reply to Hengqi Chen

The current implementation of bpf_arch_text_poke() requires 5 nops at patch site which is not applicable for kernel/module functions. With CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y, this can be done by ftrace instead.

See the following commit for details:
* commit b91e014f078e ("bpf: Make BPF trampoline use register_ftrace_direct() API")
* commit 9cdc3b6a299c ("LoongArch: ftrace: Add direct call support")

---
 arch/loongarch/net/bpf_jit.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +5 -2
index 7b7e449b9ea9..35b13d91a979 100644
@@ -1294,8 +1294,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
 	u32 old_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
 	u32 new_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
 
-	if (!is_kernel_text((unsigned long)ip) &&
-		!is_bpf_text_address((unsigned long)ip))
+	if (!is_bpf_text_address((unsigned long)ip))
+		/* Only poking bpf text is supported. Since kernel function
+		 * entry is set up by ftrace, we reply on ftrace to poke kernel
+		 * functions.
+		 */
 		return -ENOTSUPP;
 
 	ret = emit_jump_or_nops(old_addr, ip, old_insns, is_call);
Signature · 1 line
2.43.5
maintainer
#9 lore raw

Re: [PATCH v4 4/8] LoongArch: BPF: No text_poke() for kernel text

Hi, Hengqi,

Quoted 5 lines · Hengqi Chen

The current implementation of bpf_arch_text_poke() requires 5 nops at patch site which is not applicable for kernel/module functions. With CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y, this can be done by ftrace instead.

Does this mean BPF trampoline can only work with FTRACE enabled?

Huacai

Quoted 31 lines

See the following commit for details:
* commit b91e014f078e ("bpf: Make BPF trampoline use register_ftrace_direct() API")
* commit 9cdc3b6a299c ("LoongArch: ftrace: Add direct call support")

---
 arch/loongarch/net/bpf_jit.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +5 -3
index 7b7e449b9ea9..35b13d91a979 100644
@@ -1294,8 +1294,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
        u32 old_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
        u32 new_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
 
-       if (!is_kernel_text((unsigned long)ip) &&
-               !is_bpf_text_address((unsigned long)ip))
+       if (!is_bpf_text_address((unsigned long)ip))
+               /* Only poking bpf text is supported. Since kernel function
+                * entry is set up by ftrace, we reply on ftrace to poke kernel
+                * functions.
+                */
                return -ENOTSUPP;
 
        ret = emit_jump_or_nops(old_addr, ip, old_insns, is_call);
--

2.43.5

author reviewer
#10 lore raw

Re: [PATCH v4 4/8] LoongArch: BPF: No text_poke() for kernel text

Quoted 11 lines · Huacai Chen

Hi, Hengqi,

Quoted 5 lines · Hengqi Chen

The current implementation of bpf_arch_text_poke() requires 5 nops at patch site which is not applicable for kernel/module functions. With CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y, this can be done by ftrace instead.

Does this mean BPF trampoline can only work with FTRACE enabled?

IIUC, with CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=n, we could still trace BPF progs with BPF trampoline.

Quoted 33 lines

Huacai

Quoted 31 lines

See the following commit for details:
* commit b91e014f078e ("bpf: Make BPF trampoline use register_ftrace_direct() API")
* commit 9cdc3b6a299c ("LoongArch: ftrace: Add direct call support")

---
 arch/loongarch/net/bpf_jit.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +5 -3
index 7b7e449b9ea9..35b13d91a979 100644
@@ -1294,8 +1294,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
        u32 old_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
        u32 new_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
 
-       if (!is_kernel_text((unsigned long)ip) &&
-               !is_bpf_text_address((unsigned long)ip))
+       if (!is_bpf_text_address((unsigned long)ip))
+               /* Only poking bpf text is supported. Since kernel function
+                * entry is set up by ftrace, we reply on ftrace to poke kernel
+                * functions.
+                */
                return -ENOTSUPP;
 
        ret = emit_jump_or_nops(old_addr, ip, old_insns, is_call);
--

2.43.5

maintainer
#11 lore raw

Re: [PATCH v4 4/8] LoongArch: BPF: No text_poke() for kernel text

Quoted 11 lines · Hengqi Chen
Quoted 9 lines · Huacai Chen

Hi, Hengqi,

Quoted 5 lines · Hengqi Chen

The current implementation of bpf_arch_text_poke() requires 5 nops at patch site which is not applicable for kernel/module functions. With CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=y, this can be done by ftrace instead.

Quoted 5 lines
Quoted 2 lines

Does this mean BPF trampoline can only work with FTRACE enabled?

IIUC, with CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS=n, we could still trace BPF progs with BPF trampoline.

I don't know, but I was confused by the commit message.

Huacai

Quoted 34 lines
Quoted 34 lines

Huacai

Quoted 31 lines

See the following commit for details:
* commit b91e014f078e ("bpf: Make BPF trampoline use register_ftrace_direct() API")
* commit 9cdc3b6a299c ("LoongArch: ftrace: Add direct call support")

---
 arch/loongarch/net/bpf_jit.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +5 -3
index 7b7e449b9ea9..35b13d91a979 100644
@@ -1294,8 +1294,11 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
        u32 old_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
        u32 new_insns[LOONGARCH_LONG_JUMP_NINSNS] = {[0 ... 4] = INSN_NOP};
 
-       if (!is_kernel_text((unsigned long)ip) &&
-               !is_bpf_text_address((unsigned long)ip))
+       if (!is_bpf_text_address((unsigned long)ip))
+               /* Only poking bpf text is supported. Since kernel function
+                * entry is set up by ftrace, we reply on ftrace to poke kernel
+                * functions.
+                */
                return -ENOTSUPP;
 
        ret = emit_jump_or_nops(old_addr, ip, old_insns, is_call);
--

2.43.5

author reviewer
#12 lore raw

PATCH v4 5/8 LoongArch: BPF: Don't assume trampoline size is page aligned

In reply to Hengqi Chen

Currently, arch_alloc_bpf_trampoline() use bpf_prog_pack_alloc() which will pack multiple trampolines into a huge page. So no need to assume the trampoline size is page aligned.

---
 arch/loongarch/net/bpf_jit.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +1 -2
index 35b13d91a979..43628b5e1553 100644
@@ -1747,8 +1747,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 
 	ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
 
-	/* Page align */
-	return ret < 0 ? ret : round_up(ret * LOONGARCH_INSN_SIZE, PAGE_SIZE);
+	return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
 }
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
Signature · 1 line
2.43.5
maintainer
#13 lore raw

Re: [PATCH v4 5/8] LoongArch: BPF: Don't assume trampoline size is page aligned

Hi, Hengqi,

Quoted 4 lines · Hengqi Chen

Currently, arch_alloc_bpf_trampoline() use bpf_prog_pack_alloc() which will pack multiple trampolines into a huge page. So no need to assume the trampoline size is page aligned.

We do the alignment because larch_insn_text_copy() changes page attrs. If there is other data and BPF trampoline is in the same page, changing page attrs may cause errors.

Huacai

Quoted 23 lines
---
 arch/loongarch/net/bpf_jit.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +1 -3
index 35b13d91a979..43628b5e1553 100644
@@ -1747,8 +1747,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 
        ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
 
-       /* Page align */
-       return ret < 0 ? ret : round_up(ret * LOONGARCH_INSN_SIZE, PAGE_SIZE);
+       return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
 }
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
--

2.43.5

author reviewer
#14 lore raw

Re: [PATCH v4 5/8] LoongArch: BPF: Don't assume trampoline size is page aligned

Quoted 12 lines · Huacai Chen

Hi, Hengqi,

Quoted 4 lines · Hengqi Chen

Currently, arch_alloc_bpf_trampoline() use bpf_prog_pack_alloc() which will pack multiple trampolines into a huge page. So no need to assume the trampoline size is page aligned.

We do the alignment because larch_insn_text_copy() changes page attrs. If there is other data and BPF trampoline is in the same page, changing page attrs may cause errors.

Doesn't stop_machine() prevent this side effect?

Quoted 25 lines

Huacai

Quoted 23 lines
---
 arch/loongarch/net/bpf_jit.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +1 -3
index 35b13d91a979..43628b5e1553 100644
@@ -1747,8 +1747,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 
        ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
 
-       /* Page align */
-       return ret < 0 ? ret : round_up(ret * LOONGARCH_INSN_SIZE, PAGE_SIZE);
+       return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
 }
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
--

2.43.5

maintainer
#15 lore raw

Re: [PATCH v4 5/8] LoongArch: BPF: Don't assume trampoline size is page aligned

Quoted 10 lines · Hengqi Chen
Quoted 8 lines · Huacai Chen

Hi, Hengqi,

Quoted 4 lines · Hengqi Chen

Currently, arch_alloc_bpf_trampoline() use bpf_prog_pack_alloc() which will pack multiple trampolines into a huge page. So no need to assume the trampoline size is page aligned.

Quoted 6 lines
Quoted 4 lines

We do the alignment because larch_insn_text_copy() changes page attrs. If there is other data and BPF trampoline is in the same page, changing page attrs may cause errors.

Doesn't stop_machine() prevent this side effect?

I don't think so.

Consider such a case: BPF trampoline and another vmalloc area share the same page, larch_insn_text_copy() makes the whole page be "rox", even if stop_machine() completes. But "rox" is only needed by BPF trampoline, and the other vmalloc area isn't.

Huacai

Quoted 27 lines
Quoted 26 lines

Huacai

Quoted 23 lines
---
 arch/loongarch/net/bpf_jit.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +1 -3
index 35b13d91a979..43628b5e1553 100644
@@ -1747,8 +1747,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
 
        ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags);
 
-       /* Page align */
-       return ret < 0 ? ret : round_up(ret * LOONGARCH_INSN_SIZE, PAGE_SIZE);
+       return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE;
 }
 
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
--

2.43.5

author reviewer
#16 lore raw

PATCH v4 6/8 LoongArch: BPF: Make trampoline size stable

In reply to Hengqi Chen

When attach fentry/fexit BPF programs, __arch_prepare_bpf_trampoline() is called twice with different struct bpf_tramp_image *im:

    bpf_trampoline_update
        -> arch_bpf_trampoline_size
            -> __arch_prepare_bpf_trampoline
        -> arch_prepare_bpf_trampoline
            -> __arch_prepare_bpf_trampoline

Use move_imm() will emit unstable instruction sequences, let's use move_addr() instead to prevent subtle bugs.

(I observed this while debugging other issues with printk.)

---
 arch/loongarch/net/bpf_jit.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/loongarch/net/bpf_jit.c +2 -2
index 43628b5e1553..07ca32c673d5 100644
@@ -1599,7 +1599,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 		orig_call += LOONGARCH_BPF_FENTRY_NBYTES;
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
-		move_imm(ctx, LOONGARCH_GPR_A0, (const s64)im, false);
+		move_addr(ctx, LOONGARCH_GPR_A0, (const u64)im);
 		ret = emit_call(ctx, (const u64)__bpf_tramp_enter);
 		if (ret)
 			return ret;
@@ -1649,7 +1649,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 
 	if (flags & BPF_TRAMP_F_CALL_ORIG) {
 		im->ip_epilogue = ctx->ro_image + ctx->idx;
-		move_imm(ctx, LOONGARCH_GPR_A0, (const s64)im, false);
+		move_addr(ctx, LOONGARCH_GPR_A0, (const u64)im);
 		ret = emit_call(ctx, (const u64)__bpf_tramp_exit);
 		if (ret)
 			goto out;
Signature · 1 line
2.43.5
author reviewer
#17 lore raw

PATCH v4 7/8 LoongArch: BPF: Make error handling robust in arch_prepare_bpf_trampoline()

In reply to Hengqi Chen

Bail out instead of trying to perform a bpf_arch_text_copy() if __arch_prepare_bpf_trampoline() failed.

---
 arch/loongarch/net/bpf_jit.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/loongarch/net/bpf_jit.c +4 -1
index 07ca32c673d5..6dba407c202f 100644
@@ -1719,7 +1719,10 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
 
 	jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image));
 	ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags);
-	if (ret > 0 && validate_code(&ctx) < 0) {
+	if (ret < 0)
+		goto out;
+
+	if (validate_code(&ctx) < 0) {
 		ret = -EINVAL;
 		goto out;
 	}
Signature · 1 line
2.43.5
author reviewer
#18 lore raw

PATCH v4 8/8 LoongArch: BPF: Sign extend struct ops return values properly

In reply to Hengqi Chen

The ns_bpf_qdisc selftest triggers a kernel panic:

    CPU 0 Unable to handle kernel paging request at virtual address 0000000000741d58, era == 90000000851b5ac0, ra == 90000000851b5aa4
    Oops[#1]:
    CPU: 0 UID: 0 PID: 449 Comm: test_progs Tainted: G           OE       6.16.0+ #3 PREEMPT(full)
    Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
    Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
    pc 90000000851b5ac0 ra 90000000851b5aa4 tp 90000001076b8000 sp 90000001076bb600
    a0 0000000000741ce8 a1 0000000000000001 a2 90000001076bb5c0 a3 0000000000000008
    a4 90000001004c4620 a5 9000000100741ce8 a6 0000000000000000 a7 0100000000000000
    t0 0000000000000010 t1 0000000000000000 t2 9000000104d24d30 t3 0000000000000001
    t4 4f2317da8a7e08c4 t5 fffffefffc002f00 t6 90000001004c4620 t7 ffffffffc61c5b3d
    t8 0000000000000000 u0 0000000000000001 s9 0000000000000050 s0 90000001075bc800
    s1 0000000000000040 s2 900000010597c400 s3 0000000000000008 s4 90000001075bc880
    s5 90000001075bc8f0 s6 0000000000000000 s7 0000000000741ce8 s8 0000000000000000
       ra: 90000000851b5aa4 __qdisc_run+0xac/0x8d8
      ERA: 90000000851b5ac0 __qdisc_run+0xc8/0x8d8
     CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
     PRMD: 00000004 (PPLV0 +PIE -PWE)
     EUEN: 00000007 (+FPE +SXE +ASXE -BTE)
     ECFG: 00071c1d (LIE=0,2-4,10-12 VS=7)
    ESTAT: 00010000 [PIL] (IS= ECode=1 EsubCode=0)
     BADV: 0000000000741d58
     PRID: 0014c010 (Loongson-64bit, Loongson-3A5000)
    Modules linked in: bpf_testmod(OE) [last unloaded: bpf_testmod(OE)]
    Process test_progs (pid: 449, threadinfo=000000009af02b3a, task=00000000e9ba4956)
    Stack : 0000000000000000 90000001075bc8ac 90000000869524a8 9000000100741ce8
            90000001075bc800 9000000100415300 90000001075bc8ac 0000000000000000
            900000010597c400 900000008694a000 0000000000000000 9000000105b59000
            90000001075bc800 9000000100741ce8 0000000000000050 900000008513000c
            9000000086936000 0000000100094d4c fffffff400676208 0000000000000000
            9000000105b59000 900000008694a000 9000000086bf0dc0 9000000105b59000
            9000000086bf0d68 9000000085147010 90000001075be788 0000000000000000
            9000000086bf0f98 0000000000000001 0000000000000010 9000000006015840
            0000000000000000 9000000086be6c40 0000000000000000 0000000000000000
            0000000000000000 4f2317da8a7e08c4 0000000000000101 4f2317da8a7e08c4
            ...
    Call Trace:
    [<90000000851b5ac0>] __qdisc_run+0xc8/0x8d8
    [<9000000085130008>] __dev_queue_xmit+0x578/0x10f0
    [<90000000853701c0>] ip6_finish_output2+0x2f0/0x950
    [<9000000085374bc8>] ip6_finish_output+0x2b8/0x448
    [<9000000085370b24>] ip6_xmit+0x304/0x858
    [<90000000853c4438>] inet6_csk_xmit+0x100/0x170
    [<90000000852b32f0>] __tcp_transmit_skb+0x490/0xdd0
    [<90000000852b47fc>] tcp_connect+0xbcc/0x1168
    [<90000000853b9088>] tcp_v6_connect+0x580/0x8a0
    [<90000000852e7738>] __inet_stream_connect+0x170/0x480
    [<90000000852e7a98>] inet_stream_connect+0x50/0x88
    [<90000000850f2814>] __sys_connect+0xe4/0x110
    [<90000000850f2858>] sys_connect+0x18/0x28
    [<9000000085520c94>] do_syscall+0x94/0x1a0
    [<9000000083df1fb8>] handle_syscall+0xb8/0x158

    Code: 4001ad80  2400873f  2400832d <240073cc> 001137ff  001133ff  6407b41f  001503cc  0280041d

    ---[ end trace 0000000000000000 ]---

The bpf_fifo_dequeue prog returns a skb which is a pointer. The pointer is treated as a 32bit value and sign extend to 64bit in epilogue. This behavior is right for most bpf prog types but wrong for struct ops which requires LoongArch ABI.

So let's sign extend struct ops return values according to the LoongArch ABI ([0]) and return value spec in function model.

[0]: https://loongson.github.io/LoongArch-Documentation/LoongArch-ELF-ABI-EN.html

Fixes 6abf17d690d8 ("LoongArch: BPF: Add struct ops support for trampoline")
---
 arch/loongarch/net/bpf_jit.c | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)
arch/loongarch/net/bpf_jit.c +31 -1
index 6dba407c202f..583a540ff7f5 100644
@@ -1451,6 +1451,32 @@ void arch_free_bpf_trampoline(void *image, unsigned int size)
 	bpf_prog_pack_free(image, size);
 }
 
+/*
+ * Sign-extend the register if necessary
+ */
+static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, u8 flags)
+{
+	if (!(flags & BTF_FMODEL_SIGNED_ARG) && size != 8)
+		return;
+
+	switch (size) {
+	case 1:
+		emit_insn(ctx, extwb, rd, rj);
+		break;
+	case 2:
+		emit_insn(ctx, extwh, rd, rj);
+		break;
+	case 4:
+		emit_insn(ctx, addiw, rd, rj, 0);
+		break;
+	case 8:
+		move_reg(ctx, rd, rj);
+		break;
+	default:
+		pr_warn("bpf_jit: invalid size %d for sign_extend\n", size);
+	}
+}
+
 static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
 					 const struct btf_func_model *m, struct bpf_tramp_links *tlinks,
 					 void *func_addr, u32 flags)
@@ -1659,8 +1685,12 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i
 		restore_args(ctx, m->nr_args, args_off);
 
 	if (save_ret) {
-		emit_insn(ctx, ldd, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -retval_off);
 		emit_insn(ctx, ldd, regmap[BPF_REG_0], LOONGARCH_GPR_FP, -(retval_off - 8));
+		if (is_struct_ops)
+			sign_extend(ctx, LOONGARCH_GPR_A0, regmap[BPF_REG_0],
+				    m->ret_size, m->ret_flags);
+		else
+			emit_insn(ctx, ldd, LOONGARCH_GPR_A0, LOONGARCH_GPR_FP, -retval_off);
 	}
 
 	emit_insn(ctx, ldd, LOONGARCH_GPR_S1, LOONGARCH_GPR_FP, -sreg_off);
Signature · 1 line
2.43.5