Skip to contentSkip to conversation

bpf: Use 64-bit writes when inlining bpf_loop

PATCH Applied Hungyu Lin 2 participants 2 messages Nov 4 – Nov 5

Subsystems (3)
lore
Share & tools
lore

Hungyu Lin

author
#0 lore raw

PATCH 1/1 bpf: Use 64-bit writes when inlining bpf_loop

The error path in inline_bpf_loop() writes -E2BIG with BPF_MOV32_IMM(), zero-extending it to 0x00000000fffffff9 instead of the 64-bit negative value returned by the helper. A caller keeping the result in a long therefore misses the error when testing ret < 0.

Use BPF_MOV64_IMM() for the error return and loop-counter initialization. Neither write needs truncation, and this avoids the extra zero extensions required by the s390 and RISC-V JITs.

Keep the helper result in a long before checking it in the existing bpf_loop test, so its over-limit case catches the wrong upper bits. Drop the now redundant architecture-specific expectations from patch_list_insert_in_front().

Fixes 1ade23711971 ("bpf: Inline calls to bpf_loop when callback is known")
---
 kernel/bpf/fixups.c                            |  4 ++--
 tools/testing/selftests/bpf/progs/bpf_loop.c   |  8 +++++---
 .../selftests/bpf/progs/verifier_patch_list.c  | 18 ------------------
 3 files changed, 7 insertions(+), 23 deletions(-)
kernel/bpf/fixups.c +2 -2
index 6b719a28be68..321624a4b2b3 100644
@@ -2783,7 +2783,7 @@ static int inline_bpf_loop(struct bpf_verifier_env *env, int position,
 	 * expected number of iterations is too big.
 	 */
 	insn_buf[cnt++] = BPF_JMP_IMM(BPF_JLE, BPF_REG_1, BPF_MAX_LOOPS, 2);
-	insn_buf[cnt++] = BPF_MOV32_IMM(BPF_REG_0, -E2BIG);
+	insn_buf[cnt++] = BPF_MOV64_IMM(BPF_REG_0, -E2BIG);
 	insn_buf[cnt++] = BPF_JMP_IMM(BPF_JA, 0, 0, 16);
 	/* spill R6, R7, R8 to use these as loop vars */
 	insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_6, r6_offset);
@@ -2791,7 +2791,7 @@ static int inline_bpf_loop(struct bpf_verifier_env *env, int position,
 	insn_buf[cnt++] = BPF_STX_MEM(BPF_DW, BPF_REG_10, BPF_REG_8, r8_offset);
 	/* initialize loop vars */
 	insn_buf[cnt++] = BPF_MOV64_REG(reg_loop_max, BPF_REG_1);
-	insn_buf[cnt++] = BPF_MOV32_IMM(reg_loop_cnt, 0);
+	insn_buf[cnt++] = BPF_MOV64_IMM(reg_loop_cnt, 0);
 	insn_buf[cnt++] = BPF_MOV64_REG(reg_loop_ctx, BPF_REG_3);
 	/* loop header,
 	 * if reg_loop_cnt >= reg_loop_max skip the loop body
tools/testing/selftests/bpf/progs/bpf_loop.c +5 -3
index 1d194455b109..7a8ae1110c0a 100644
@@ -66,14 +66,16 @@ SEC("fentry/" SYS_PREFIX "sys_nanosleep")
 int test_prog(void *ctx)
 {
 	struct callback_ctx data = {};
+	long ret;
 
 	if (bpf_get_current_pid_tgid() >> 32 != pid)
 		return 0;
 
-	nr_loops_returned = bpf_loop(nr_loops, callback, &data, 0);
+	ret = bpf_loop(nr_loops, callback, &data, 0);
+	nr_loops_returned = ret;
 
-	if (nr_loops_returned < 0)
-		err = nr_loops_returned;
+	if (ret < 0)
+		err = ret;
 	else
 		g_output = data.output;
 
tools/testing/selftests/bpf/progs/verifier_patch_list.c +0 -18
index a9ad1dc56c3a..581b75b7b5c5 100644
@@ -298,23 +298,6 @@ __arch_arm64
 __arch_riscv64
 __arch_loongarch
 __success
-/* The verifier adds two zero extensions for these JITs. */
-#if defined(__TARGET_ARCH_s390) || defined(__TARGET_ARCH_riscv)
-__xlated("25: call pc+7")
-__xlated("...")
-__xlated("32: exit")
-__xlated("33: r0 = ")
-__xlated("35: r0 = &(void __percpu *)(r0)")
-__xlated("36: r0 = *(u64 *)(r0 +0)")
-__xlated("37: if r0 != 0x0 goto pc+2")
-__xlated("38: r0 = 1")
-__xlated("39: exit")
-__xlated("40: r1 += -1")
-__xlated("...")
-__xlated("51: if r1 != 0x0 goto pc-12")
-__xlated("52: r0 = 0")
-__xlated("53: exit")
-#else
 __xlated("23: call pc+7")
 __xlated("...")
 __xlated("30: exit")
@@ -329,7 +312,6 @@ __xlated("...")
 __xlated("49: if r1 != 0x0 goto pc-12")
 __xlated("50: r0 = 0")
 __xlated("51: exit")
-#endif
 __naked void patch_list_insert_in_front(void)
 {
 	asm volatile (

base-commit: 8fb43ffc3327d12d57d3fbe6f4820b5a522de0d1

Signature · 1 line
2.43.0
#1 lore raw
This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This patch was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 10 lines · On Wed, 4 Nov 2026 03:27:00 +0000 you wrote:

The error path in inline_bpf_loop() writes -E2BIG with BPF_MOV32_IMM(), zero-extending it to 0x00000000fffffff9 instead of the 64-bit negative value returned by the helper. A caller keeping the result in a long therefore misses the error when testing ret < 0.

Use BPF_MOV64_IMM() for the error return and loop-counter initialization. Neither write needs truncation, and this avoids the extra zero extensions required by the s390 and RISC-V JITs.

[...]

Here is the summary with links:
- [bpf-next] bpf: Use 64-bit writes when inlining bpf_loop

    https://git.kernel.org/bpf/bpf-next/c/8424858a5de0

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html