Skip to contentSkip to conversation

bpf, riscv: Add support for indirect jumps

PATCH Applied Chen Pei 3 participants 7 messages Sep 28 – Oct 11

Subsystems (6)
Kselftest BPF RISC-V BPF selftests +2 more subsystems
lore
Share & tools
lore

Chen Pei

author
#0 lore raw

v3 cover bpf, riscv: Add support for indirect jumps

The indirect jump instruction (BPF_JMP | BPF_JA | BPF_X, "gotox") and the BPF_MAP_TYPE_INSN_ARRAY jump tables it consumes are core features: the verifier accepts them on every architecture, and the x86-64, arm64 and powerpc JITs implement them. On riscv64 a program using gotox passes verification and then fails to load, because the JIT does not know the opcode and CONFIG_BPF_JIT_ALWAYS_ON leaves no interpreter to fall back onto.

This series adds the riscv64 support and turns the existing selftests on for that architecture.

Patch 1 emits "jalr zero, rd, 0" for gotox and publishes the xlated to jitted offsets through bpf_prog_update_insn_ptrs(), which is what fills in the jump table addresses. Patches 2 to 4 widen the arch guards in verifier_gotox.c, compute_live_registers.c and bpf_insn_array.c to riscv64, the same way the other architectures were enabled.

Changes since v2:
- Pick up Pu Lehui's Reviewed-by on patch 1.
- Enable two more gotox test files for riscv64: compute_live_registers.c and bpf_insn_array.c.

Changes since v1:
- Rebased onto bpf-next.
- Patch 1: reword the code comment to say "shift ctx->offset[] by one".
- Patch 2: collapse the multi-line #endif marker to a single line.
- Added Reviewed-by/Acked-by from Björn Töpel.

Testing
=======

Environment: QEMU virt rv64, with CONFIG_BPF_JIT=y, CONFIG_BPF_JIT_ALWAYS_ON=y, CONFIG_DEBUG_INFO_BTF=y; selftests cross-built with clang 23.

- test_progs -t verifier_gotox: 33/33 subtests pass.
- test_progs -t bpf_insn_array: 20/20 subtests pass.
- test_progs -t compute_live_registers: 20/20 subtests pass, including the gotox liveness subtest.
- test_progs-cpuv4 -t bpf_gotox: 14/14 subtests pass.

Chen Pei (4):
bpf, riscv: Add support for indirect jumps selftests/bpf: Enable gotox tests for riscv64 selftests/bpf: Enable gotox liveness test for riscv64 selftests/bpf: Enable instruction array test for riscv64

 arch/riscv/net/bpf_jit_comp64.c                  |  5 +++++
 arch/riscv/net/bpf_jit_core.c                    | 16 ++++++++++++++--
 .../selftests/bpf/prog_tests/bpf_insn_array.c    |  3 ++-
 .../selftests/bpf/progs/compute_live_registers.c |  5 +++--
 .../testing/selftests/bpf/progs/verifier_gotox.c |  6 ++++--
 5 files changed, 28 insertions(+), 7 deletions(-)
Signature · 1 line
2.50.1
author
#1 lore raw

PATCH v3 1/4 bpf, riscv: Add support for indirect jumps

Implement JIT support for the indirect jump instruction (BPF_JMP | BPF_JA | BPF_X), a.k.a. gotox, which lets a BPF program jump through a BPF_MAP_TYPE_INSN_ARRAY jump table.

Emit "jalr zero, rd, 0" and hand the xlated to jitted offsets to bpf_prog_update_insn_ptrs(), which is what fills in the jump table entries; without that call the load fails with -EFAULT in bpf_insn_array_ready(). ctx->offset[] holds the offset of the insn *following* insn i, as bpf_prog_fill_jited_linfo() expects, so it is shifted by one and offset[0] comes from the prologue length. build_body() now records both halves of a multi-insn record, so no slot keeps a fabricated offset.

Only the RV64 JIT is covered; RV32 keeps failing to load as before.

---

Changes since v2:
- Pick up Pu Lehui's Reviewed-by; no code change.

Changes since v1:
- Reword the code comment to say "shift ctx->offset[] by one".

 arch/riscv/net/bpf_jit_comp64.c |  5 +++++
 arch/riscv/net/bpf_jit_core.c   | 16 ++++++++++++++--
 2 files changed, 19 insertions(+), 2 deletions(-)
arch/riscv/net/bpf_jit_comp64.c +5 -0
index ed0a6f871dea..9de3749fb268 100644
@@ -1691,6 +1691,11 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_zextw(rd, rd, ctx);
 		break;
 
+	/* JUMP reg */
+	case BPF_JMP | BPF_JA | BPF_X:
+		emit_jalr(RV_REG_ZERO, rd, 0, ctx);
+		break;
+
 	/* JUMP off */
 	case BPF_JMP | BPF_JA:
 	case BPF_JMP32 | BPF_JA:
arch/riscv/net/bpf_jit_core.c +14 -2
index 470a6ace5662..5265dd5bd39e 100644
@@ -26,10 +26,13 @@ static int build_body(struct rv_jit_context *ctx, bool extra_pass, int *offset)
 		int ret;
 
 		ret = bpf_jit_emit_insn(insn, ctx, extra_pass);
-		if (ret > 0)
-			i++; /* skip the next instruction */
 		if (offset)
 			offset[i] = ctx->ninsns;
+		if (ret > 0) {
+			i++; /* skip the next instruction */
+			if (offset)
+				offset[i] = ctx->ninsns;
+		}
 		if (ret < 0)
 			return ret;
 	}
@@ -176,6 +179,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
 		for (i = 0; i < prog->len; i++)
 			ctx->offset[i] = ninsns_rvoff(ctx->offset[i]);
 		bpf_prog_fill_jited_linfo(prog, ctx->offset);
+
+		/*
+		 * bpf_prog_update_insn_ptrs() wants the start of each insn, so
+		 * shift ctx->offset[] by one and get insn 0 from the prologue.
+		 */
+		for (i = prog->len - 1; i > 0; i--)
+			ctx->offset[i] = ctx->offset[i - 1];
+		ctx->offset[0] = ninsns_rvoff(ctx->prologue_len);
+		bpf_prog_update_insn_ptrs(prog, ctx->offset, jit_data->ro_image);
 out_offset:
 		kvfree(ctx->offset);
 		kfree(jit_data);
Signature · 1 line
2.50.1
author
#2 lore raw

PATCH v3 2/4 selftests/bpf: Enable gotox tests for riscv64

In reply to Chen Pei

The riscv64 JIT now supports the gotox instruction and jump tables, so run the tests in verifier_gotox.c on riscv64 too, mirroring what was done for arm64 and powerpc.

The guard is 64-bit only because the RV32 JIT does not implement gotox.

---

Changes since v2:
- Unchanged.

Changes since v1:
- Collapse the multi-line #endif marker to a single line.

 tools/testing/selftests/bpf/progs/verifier_gotox.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/verifier_gotox.c +4 -2
index f5a9878c7b8d..96a9d914930e 100644
@@ -6,7 +6,9 @@
 #include "bpf_misc.h"
 #include "../../../include/linux/filter.h"
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || defined(__TARGET_ARCH_powerpc)
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+	defined(__TARGET_ARCH_powerpc) || \
+	(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)
 
 #define DEFINE_SIMPLE_JUMP_TABLE_PROG(NAME, SRC_REG, OFF, IMM, OUTCOME)	\
 									\
@@ -580,6 +582,6 @@ nop_%=:								\
 	: __clobber_all);
 }
 
-#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 || __TARGET_ARCH_powerpc*/
+#endif /* gotox: x86, arm64, powerpc, riscv64 */
 
 char _license[] SEC("license") = "GPL";
Signature · 1 line
2.50.1
author
#3 lore raw

PATCH v3 3/4 selftests/bpf: Enable gotox liveness test for riscv64

In reply to Chen Pei

The riscv64 JIT now supports gotox, so enable the gotox liveness test in compute_live_registers.c on riscv64 as well.

The guard is 64-bit only because the RV32 JIT does not implement gotox.

---

 tools/testing/selftests/bpf/progs/compute_live_registers.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/progs/compute_live_registers.c +3 -2
index 941b65fe7c07..e482c10004a0 100644
@@ -467,7 +467,8 @@ __naked void subprog_ret_reg_pair(void)
 		::: __clobber_all);
 }
 
-#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64)
+#if defined(__TARGET_ARCH_x86) || defined(__TARGET_ARCH_arm64) || \
+	(defined(__TARGET_ARCH_riscv) && __riscv_xlen == 64)
 
 SEC("socket")
 __log_level(2)
@@ -506,7 +507,7 @@ void gotox(void)
 	: __clobber_all);
 }
 
-#endif /* __TARGET_ARCH_x86 || __TARGET_ARCH_arm64 */
+#endif /* gotox: x86, arm64, riscv64 */
 
 /* to retain debug info for BTF generation */
 void kfunc_root(void)
Signature · 1 line
2.50.1
author
#4 lore raw

PATCH v3 4/4 selftests/bpf: Enable instruction array test for riscv64

In reply to Chen Pei

As the riscv64 JIT now supports instruction array (gotox jump tables), enable the corresponding bpf selftest for riscv64. The guard is 64-bit only because the RV32 JIT does not implement gotox.

---

 tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/bpf_insn_array.c +2 -1
index bfedeed24ac5..4ffe4fe6b78e 100644
@@ -4,7 +4,8 @@
 #include <bpf/btf.h>
 #include <test_progs.h>
 
-#if defined(__x86_64__) || defined(__powerpc__) || defined(__aarch64__)
+#if defined(__x86_64__) || defined(__powerpc__) || defined(__aarch64__) || \
+	(defined(__riscv) && __riscv_xlen == 64)
 static int map_create(__u32 map_type, __u32 max_entries)
 {
 	const char *map_name = "insn_array";
Signature · 1 line
2.50.1
maintainer Tested-by on the v3 series
#5 lore raw
In reply to Chen Pei
Quoted 47 lines · Chen Pei

The indirect jump instruction (BPF_JMP | BPF_JA | BPF_X, "gotox") and the BPF_MAP_TYPE_INSN_ARRAY jump tables it consumes are core features: the verifier accepts them on every architecture, and the x86-64, arm64 and powerpc JITs implement them. On riscv64 a program using gotox passes verification and then fails to load, because the JIT does not know the opcode and CONFIG_BPF_JIT_ALWAYS_ON leaves no interpreter to fall back onto.

This series adds the riscv64 support and turns the existing selftests on for that architecture.

Patch 1 emits "jalr zero, rd, 0" for gotox and publishes the xlated to jitted offsets through bpf_prog_update_insn_ptrs(), which is what fills in the jump table addresses. Patches 2 to 4 widen the arch guards in verifier_gotox.c, compute_live_registers.c and bpf_insn_array.c to riscv64, the same way the other architectures were enabled.

Changes since v2:
- Pick up Pu Lehui's Reviewed-by on patch 1.
- Enable two more gotox test files for riscv64: compute_live_registers.c and bpf_insn_array.c.

Changes since v1:
- Rebased onto bpf-next.
- Patch 1: reword the code comment to say "shift ctx->offset[] by one".
- Patch 2: collapse the multi-line #endif marker to a single line.
- Added Reviewed-by/Acked-by from Björn Töpel.

Testing
=======

Environment: QEMU virt rv64, with CONFIG_BPF_JIT=y, CONFIG_BPF_JIT_ALWAYS_ON=y, CONFIG_DEBUG_INFO_BTF=y; selftests cross-built with clang 23.

- test_progs -t verifier_gotox: 33/33 subtests pass.
- test_progs -t bpf_insn_array: 20/20 subtests pass.
- test_progs -t compute_live_registers: 20/20 subtests pass, including the gotox liveness subtest.
- test_progs-cpuv4 -t bpf_gotox: 14/14 subtests pass.

Chen Pei (4):
bpf, riscv: Add support for indirect jumps selftests/bpf: Enable gotox tests for riscv64 selftests/bpf: Enable gotox liveness test for riscv64 selftests/bpf: Enable instruction array test for riscv64

Quoted 7 lines
  arch/riscv/net/bpf_jit_comp64.c                  |  5 +++++
  arch/riscv/net/bpf_jit_core.c                    | 16 ++++++++++++++--
  .../selftests/bpf/prog_tests/bpf_insn_array.c    |  3 ++-
  .../selftests/bpf/progs/compute_live_registers.c |  5 +++--
  .../testing/selftests/bpf/progs/verifier_gotox.c |  6 ++++--
  5 files changed, 28 insertions(+), 7 deletions(-)

For this series:

#6 lore raw
In reply to Chen Pei
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 9 lines · On Mon, 28 Sep 2026 15:18:41 +0800 you wrote:

The indirect jump instruction (BPF_JMP | BPF_JA | BPF_X, "gotox") and the BPF_MAP_TYPE_INSN_ARRAY jump tables it consumes are core features: the verifier accepts them on every architecture, and the x86-64, arm64 and powerpc JITs implement them. On riscv64 a program using gotox passes verification and then fails to load, because the JIT does not know the opcode and CONFIG_BPF_JIT_ALWAYS_ON leaves no interpreter to fall back onto.

[...]

Here is the summary with links:
- [bpf-next,v3,1/4] bpf, riscv: Add support for indirect jumps

    https://git.kernel.org/bpf/bpf-next/c/c0316386fecd
  - [bpf-next,v3,2/4] selftests/bpf: Enable gotox tests for riscv64
    https://git.kernel.org/bpf/bpf-next/c/5f750ad6d675
  - [bpf-next,v3,3/4] selftests/bpf: Enable gotox liveness test for riscv64
    https://git.kernel.org/bpf/bpf-next/c/6ce75af4f59e
  - [bpf-next,v3,4/4] selftests/bpf: Enable instruction array test for riscv64
    https://git.kernel.org/bpf/bpf-next/c/eec2444d9b9d

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html