Skip to contentSkip to conversation

bpf,riscv: Add support for BPF Arena

PATCH Superseded Puranjay Mohan 3 participants 6 messages Mar 26, 2024 – Mar 27, 2024

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover bpf,riscv: Add support for BPF Arena

Changes in V3
V2: https://lore.kernel.org/bpf/[email protected]/
- Use REG_DONT_CLEAR_MARKER in place of DONT_CLEAR as the name.
- Use RV_REG_ZERO for the above as it is guaranteed to be never used in as dst/src register in BPF LDX/STX/ST instructions.
- Use #define RV_REG_ARENA for RV_REG_S7

Changes in V2
V1: https://lore.kernel.org/bpf/[email protected]/
- Used S7 in place of S11 for storing the kernel_vm_start
- Used 17 in place of 16 for DONT_CLEAR marker
- Remove an unused variable
- Removed some misleading information from the commit message.

This series adds the support for PROBE_MEM32 and bpf_addr_space_cast instructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.

All arena related selftests are passing:

  root@rv-tester:~/bpf# uname -p
  riscv64
  root@rv-tester:~/bpf# ./test_progs -a "*arena*"
  #3/1     arena_htab/arena_htab_llvm:OK
  #3/2     arena_htab/arena_htab_asm:OK
  #3       arena_htab:OK
  #4/1     arena_list/arena_list_1:OK
  #4/2     arena_list/arena_list_1000:OK
  #4       arena_list:OK
  #434/1   verifier_arena/basic_alloc1:OK
  #434/2   verifier_arena/basic_alloc2:OK
  #434/3   verifier_arena/basic_alloc3:OK
  #434/4   verifier_arena/iter_maps1:OK
  #434/5   verifier_arena/iter_maps2:OK
  #434/6   verifier_arena/iter_maps3:OK
  #434     verifier_arena:OK
  Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED

This feature needs the following two fixes in the bpf/bpf.git to work properly:

f7f5d1808b1b6 ("bpf: verifier: fix addr_space_cast from as(1) to as(0)") 443574b033876 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")

There is a lot of code repetition for LDX, STX, and ST. I will be sending a follow-up patch to refactor these.

Puranjay Mohan (2):
bpf,riscv: Implement PROBE_MEM32 pseudo instructions bpf,riscv: Implement bpf_addr_space_cast instruction

 arch/riscv/net/bpf_jit.h        |   2 +
 arch/riscv/net/bpf_jit_comp64.c | 205 +++++++++++++++++++++++++++++++-
 arch/riscv/net/bpf_jit_core.c   |   2 +
 3 files changed, 207 insertions(+), 2 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH v3 1/2 bpf,riscv: Implement PROBE_MEM32 pseudo instructions

Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences:
- PROBE_MEM32 supports store.
- PROBE_MEM32 relies on the verifier to clear upper 32-bit of the src/dst register
- PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in S7 in the prologue). Due to bpf_arena constructions such S7 + reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time.
- S11 is a free callee-saved register, so it is used to store kern_vm_start
- PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed.

To support these on riscv, we do tmp = S7 + src/dst reg and then use tmp2 as the new src/dst register. This allows us to reuse most of the code for normal [LDX | STX | ST].

---
 arch/riscv/net/bpf_jit.h        |   1 +
 arch/riscv/net/bpf_jit_comp64.c | 191 +++++++++++++++++++++++++++++++-
 arch/riscv/net/bpf_jit_core.c   |   1 +
 3 files changed, 191 insertions(+), 2 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
index f4b6b3b9edda..8a47da08dd9c 100644
@@ -81,6 +81,7 @@ struct rv_jit_context {
 	int nexentries;
 	unsigned long flags;
 	int stack_size;
+	u64 arena_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +189 -2

263 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +1 -0
index 6b3acac30c06..9ab739b9f9a2 100644
@@ -80,6 +80,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		goto skip_init_ctx;
 	}
 
+	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
 	if (!ctx->offset) {
Signature · 1 line
2.40.1
Reviewed-by Tested-by on v3 1/2
#2 lore raw

Re: [PATCH bpf-next v3 1/2] bpf,riscv: Implement PROBE_MEM32 pseudo instructions

Quoted 1 line · Puranjay Mohan

Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW]

[SNIP]

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1550

Quoted 6 lines

if (WARN_ON_ONCE(ctx->nexentries >= ctx->prog->aux->num_exentries))

arch/riscv/net/bpf_jit_comp64.c +1 -0
@@ -1539,6 +1547,11 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	case BPF_LDX | BPF_PROBE_MEMSX | BPF_B:
  	case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
  	case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
+	/* LDX | PROBE_MEM32: dst = *(unsigned size *)(src + S7 + off)*/

still some nits. please align, change S7 here to reg_arena.

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+	case BPF_LDX | BPF_PROBE_MEM32 | BPF_B:
+	case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
+	case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
+	case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:

[SNIP]

Quoted 6 lines
arch/riscv/net/bpf_jit_comp64.c +6 -0
+
+		switch (BPF_SIZE(code)) {
+		case BPF_B:
+			if (is_12b_int(off)) {
+				insns_start = ctx->ninsns;
+				emit(rv_sb(rd, off, RV_REG_T1), ctx);

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1730

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+				insn_len = ctx->ninsns - insns_start;
+				break;
+			}
+
+			emit_imm(RV_REG_T2, off, ctx);
+			emit_add(RV_REG_T2, RV_REG_T2, rd, ctx);
+			insns_start = ctx->ninsns;
+			emit(rv_sb(RV_REG_T2, 0, RV_REG_T1), ctx);
+			insn_len = ctx->ninsns - insns_start;
+

redundant blank.

Review

Quoted 2 lines
(quoted diff) +2 -0
+			break;
+

ditto.

Others, looks good to me.

#3 lore raw

PATCH v3 2/2 bpf,riscv: Implement bpf_addr_space_cast instruction

In reply to Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.

rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY

rY = addr_space_cast(rX, 1, 0) has to be converted by JIT.

---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 14 ++++++++++++++
 arch/riscv/net/bpf_jit_core.c   |  1 +
 3 files changed, 16 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
index 8a47da08dd9c..5fc374ed98ea 100644
@@ -82,6 +82,7 @@ struct rv_jit_context {
 	unsigned long flags;
 	int stack_size;
 	u64 arena_vm_start;
+	u64 user_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +14 -0
index 73726f455f6b..77ea306452d4 100644
@@ -1081,6 +1081,15 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 	/* dst = src */
 	case BPF_ALU | BPF_MOV | BPF_X:
 	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (insn_is_cast_user(insn)) {
+			emit_mv(RV_REG_T1, rs, ctx);
+			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);
+			break;
+		}
 		if (imm == 1) {
 			/* Special mov32 for zext */
 			emit_zextw(rd, rd, ctx);
@@ -2026,3 +2035,8 @@ bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_arena(void)
+{
+	return true;
+}
arch/riscv/net/bpf_jit_core.c +1 -0
index 9ab739b9f9a2..8a69d6d81e32 100644
@@ -81,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	}
 
 	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
+	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
 	if (!ctx->offset) {
Signature · 1 line
2.40.1
Reviewed-by Tested-by on v3 2/2
#4 lore raw

Re: [PATCH bpf-next v3 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

Quoted 63 lines · Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.

rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY

rY = addr_space_cast(rX, 1, 0) has to be converted by JIT.

---
  arch/riscv/net/bpf_jit.h        |  1 +
  arch/riscv/net/bpf_jit_comp64.c | 14 ++++++++++++++
  arch/riscv/net/bpf_jit_core.c   |  1 +
  3 files changed, 16 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
index 8a47da08dd9c..5fc374ed98ea 100644
@@ -82,6 +82,7 @@ struct rv_jit_context {
  	unsigned long flags;
  	int stack_size;
  	u64 arena_vm_start;
+	u64 user_vm_start;
  };
 
  /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +14 -0
index 73726f455f6b..77ea306452d4 100644
@@ -1081,6 +1081,15 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	/* dst = src */
  	case BPF_ALU | BPF_MOV | BPF_X:
  	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (insn_is_cast_user(insn)) {
+			emit_mv(RV_REG_T1, rs, ctx);
+			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);
+			break;
+		}
  		if (imm == 1) {
  			/* Special mov32 for zext */
  			emit_zextw(rd, rd, ctx);
@@ -2026,3 +2035,8 @@ bool bpf_jit_supports_ptr_xchg(void)
  {
  	return true;
  }
+
+bool bpf_jit_supports_arena(void)
+{
+	return true;
+}
arch/riscv/net/bpf_jit_core.c +0 -0
index 9ab739b9f9a2..8a69d6d81e32 100644

arch/riscv/net/bpf_jit_core.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 87

Quoted 7 lines
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -81,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	}
 
  	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
+	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
  	ctx->prog = prog;
  	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
  	if (!ctx->offset) {
maintainer Tested-by Acked-by on the v3 series
#5 lore raw
In reply to Puranjay Mohan
Quoted 39 lines · Puranjay Mohan

Changes in V3
V2: https://lore.kernel.org/bpf/[email protected]/
- Use REG_DONT_CLEAR_MARKER in place of DONT_CLEAR as the name.
- Use RV_REG_ZERO for the above as it is guaranteed to be never used in as dst/src register in BPF LDX/STX/ST instructions.
- Use #define RV_REG_ARENA for RV_REG_S7

Changes in V2
V1: https://lore.kernel.org/bpf/[email protected]/
- Used S7 in place of S11 for storing the kernel_vm_start
- Used 17 in place of 16 for DONT_CLEAR marker
- Remove an unused variable
- Removed some misleading information from the commit message.

This series adds the support for PROBE_MEM32 and bpf_addr_space_cast instructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.

All arena related selftests are passing:

  root@rv-tester:~/bpf# uname -p
  riscv64
  root@rv-tester:~/bpf# ./test_progs -a "*arena*"
  #3/1     arena_htab/arena_htab_llvm:OK
  #3/2     arena_htab/arena_htab_asm:OK
  #3       arena_htab:OK
  #4/1     arena_list/arena_list_1:OK
  #4/2     arena_list/arena_list_1000:OK
  #4       arena_list:OK
  #434/1   verifier_arena/basic_alloc1:OK
  #434/2   verifier_arena/basic_alloc2:OK
  #434/3   verifier_arena/basic_alloc3:OK
  #434/4   verifier_arena/iter_maps1:OK
  #434/5   verifier_arena/iter_maps2:OK
  #434/6   verifier_arena/iter_maps3:OK
  #434     verifier_arena:OK
  Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED
Quoted 8 lines

This feature needs the following two fixes in the bpf/bpf.git to work properly:

f7f5d1808b1b6 ("bpf: verifier: fix addr_space_cast from as(1) to as(0)") 443574b033876 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")

There is a lot of code repetition for LDX, STX, and ST. I will be sending a follow-up patch to refactor these.

Thank you, and please address Lehui's remarks as well.

For the series: