Skip to contentSkip to conversation

bpf,riscv: Add support for BPF Arena

PATCH Superseded Puranjay Mohan 3 participants 8 messages Mar 23, 2024 – Mar 24, 2024

Subsystems (4)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

cover bpf,riscv: Add support for BPF Arena

This series adds the support for PROBE_MEM32 and bpf_addr_space_cast instructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.

All arena related selftests are passing:

  root@rv-tester:~/bpf# uname -p
  riscv64
  root@rv-tester:~/bpf# ./test_progs -a "*arena*"
  #3/1     arena_htab/arena_htab_llvm:OK
  #3/2     arena_htab/arena_htab_asm:OK
  #3       arena_htab:OK
  #4/1     arena_list/arena_list_1:OK
  #4/2     arena_list/arena_list_1000:OK
  #4       arena_list:OK
  #434/1   verifier_arena/basic_alloc1:OK
  #434/2   verifier_arena/basic_alloc2:OK
  #434/3   verifier_arena/basic_alloc3:OK
  #434/4   verifier_arena/iter_maps1:OK
  #434/5   verifier_arena/iter_maps2:OK
  #434/6   verifier_arena/iter_maps3:OK
  #434     verifier_arena:OK
  Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED

This needs the following commit in bpf/bpf.git to work correctly:
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=f7f5d1808b1b66935a24dd796dd1a0612ca9c147

Puranjay Mohan (2):
bpf,riscv: Implement PROBE_MEM32 pseudo instructions bpf,riscv: Implement bpf_addr_space_cast instruction

 arch/riscv/net/bpf_jit.h        |   2 +
 arch/riscv/net/bpf_jit_comp64.c | 208 +++++++++++++++++++++++++++++++-
 arch/riscv/net/bpf_jit_core.c   |   3 +
 3 files changed, 210 insertions(+), 3 deletions(-)
Signature · 1 line
2.40.1
#1 lore raw

PATCH 1/2 bpf,riscv: Implement PROBE_MEM32 pseudo instructions

Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences:
- PROBE_MEM32 supports store.
- PROBE_MEM32 relies on the verifier to clear upper 32-bit of the src/dst register
- PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in S11 in the prologue). Due to bpf_arena constructions such S11 + reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time.
- S11 is a free callee-saved register, so it is used to store kern_vm_start
- PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed.

To support these on riscv, we do tmp = S11 + src/dst reg and then use tmp2 as the new src/dst register. This allows us to reuse most of the code for normal [LDX | STX | ST].

---
 arch/riscv/net/bpf_jit.h        |   1 +
 arch/riscv/net/bpf_jit_comp64.c | 193 +++++++++++++++++++++++++++++++-
 arch/riscv/net/bpf_jit_core.c   |   2 +
 3 files changed, 193 insertions(+), 3 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
index f4b6b3b9edda..8a47da08dd9c 100644
@@ -81,6 +81,7 @@ struct rv_jit_context {
 	int nexentries;
 	unsigned long flags;
 	int stack_size;
+	u64 arena_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +190 -3

271 lines not on this page · raw message

arch/riscv/net/bpf_jit_core.c +2 -0
index 6b3acac30c06..9b6696b1290a 100644
@@ -50,6 +50,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	int pass = 0, prev_ninsns = 0, i;
 	struct rv_jit_data *jit_data;
 	struct rv_jit_context *ctx;
+	u64 arena_vm_start;
 
 	if (!prog->jit_requested)
 		return orig_prog;
@@ -80,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 		goto skip_init_ctx;
 	}
 
+	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
 	if (!ctx->offset) {
Signature · 1 line
2.40.1
#3 lore raw

PATCH 2/2 bpf,riscv: Implement bpf_addr_space_cast instruction

In reply to Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.

rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY

rY = addr_space_cast(rX, 1, 0) has to be converted by JIT:

Here I explain using symbolic language what the JIT is supposed to do:
We have:

	src = [src_upper32][src_lower32] // 64 bit src kernel pointer
	uvm = [uvm_upper32][uvm_lower32] // 64 bit user_vm_start

The JIT has to make the dst reg like following

	dst = [uvm_upper32][src_lower32] // if src_lower32 != 0
	dst = [00000000000][00000000000] // if src_lower32 == 0
---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++
 arch/riscv/net/bpf_jit_core.c   |  1 +
 3 files changed, 17 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
index 8a47da08dd9c..5fc374ed98ea 100644
@@ -82,6 +82,7 @@ struct rv_jit_context {
 	unsigned long flags;
 	int stack_size;
 	u64 arena_vm_start;
+	u64 user_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +15 -0
index f51b832eafb6..3c389e75cb96 100644
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 	/* dst = src */
 	case BPF_ALU | BPF_MOV | BPF_X:
 	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+		    insn->imm == 1U << 16) {
+			emit_mv(RV_REG_T1, rs, ctx);
+			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);
+			break;
+		}
 		if (imm == 1) {
 			/* Special mov32 for zext */
 			emit_zextw(rd, rd, ctx);
@@ -2010,3 +2020,8 @@ bool bpf_jit_supports_ptr_xchg(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_arena(void)
+{
+	return true;
+}
arch/riscv/net/bpf_jit_core.c +1 -0
index 9b6696b1290a..aaef1d0c7c46 100644
@@ -82,6 +82,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
 	}
 
 	ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
+	ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
 	ctx->prog = prog;
 	ctx->offset = kcalloc(prog->len, sizeof(int), GFP_KERNEL);
 	if (!ctx->offset) {
Signature · 1 line
2.40.1
maintainer
#2 lore raw

Re: [PATCH bpf-next 1/2] bpf,riscv: Implement PROBE_MEM32 pseudo instructions

In reply to Puranjay Mohan
Quoted 1 line · Puranjay Mohan

Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW]

[snip]

Quoted 4 lines

#define BPF_FIXUP_OFFSET_MASK GENMASK(26, 0)
#define BPF_FIXUP_REG_MASK GENMASK(31, 27)
+#define DONT_CLEAR 16 /* RV_REG_A6 unused in BPF */

This is a bit misleading. RV_REG_A6 is actually used in riscv64. Maybe "RV_REG_A6 unused in pt_regmap" or change to other register will be better.

Quoted 2 lines

bool ex_handler_bpf(const struct exception_table_entry *ex,

[snip]

arch/riscv/net/bpf_jit_comp64.c void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) · line 1980

Quoted 11 lines
  
  	stack_adjust = round_up(stack_adjust, 16);
  	stack_adjust += bpf_stack_adjust;
arch/riscv/net/bpf_jit_comp64.c +4 -0
@@ -1794,6 +1974,10 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
  		emit_sd(RV_REG_SP, store_offset, RV_REG_S6, ctx);
  		store_offset -= 8;
  	}
+	if (ctx->arena_vm_start) {
+		emit_sd(RV_REG_SP, store_offset, RV_REG_S11, ctx);
+		store_offset -= 8;
+	}

I think it's fine to use s7 and keep the original dynamic stack code style.

Quoted 18 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
  	emit_addi(RV_REG_FP, RV_REG_SP, stack_adjust, ctx);
 
@@ -1807,6 +1991,9 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
  		emit_mv(RV_REG_TCC_SAVED, RV_REG_TCC, ctx);
 
  	ctx->stack_size = stack_adjust;
+
+	if (ctx->arena_vm_start)
+		emit_imm(RV_REG_S11, ctx->arena_vm_start, ctx);
  }
 
  void bpf_jit_build_epilogue(struct rv_jit_context *ctx)
arch/riscv/net/bpf_jit_core.c +0 -0
index 6b3acac30c06..9b6696b1290a 100644

arch/riscv/net/bpf_jit_core.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 53

Quoted 4 lines
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -50,6 +50,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
  	int pass = 0, prev_ninsns = 0, i;
  	struct rv_jit_data *jit_data;
  	struct rv_jit_context *ctx;
+	u64 arena_vm_start;

unused variable

maintainer
#4 lore raw

Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

In reply to Puranjay Mohan
Quoted 1 line · Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating

[snip]

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088

Quoted 13 lines

/* Convert from ninsns to bytes. */

arch/riscv/net/bpf_jit_comp64.c +3 -0
index f51b832eafb6..3c389e75cb96 100644
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	/* dst = src */
  	case BPF_ALU | BPF_MOV | BPF_X:
  	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+		    insn->imm == 1U << 16) {
+			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);

combine mv and zextw will be better

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);

ditto, but for or and mv

#5 lore raw

Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

Quoted 4 lines · Pu Lehui
Quoted 1 line · Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating

[snip]

Quoted 1 line
Quoted 13 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088

Quoted 12 lines

/* Convert from ninsns to bytes. */

arch/riscv/net/bpf_jit_comp64.c +3 -0
index f51b832eafb6..3c389e75cb96 100644
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
  	/* dst = src */
  	case BPF_ALU | BPF_MOV | BPF_X:
  	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+		    insn->imm == 1U << 16) {
+			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);

combine mv and zextw will be better

Do you suggest doing:

emit_zextw(RV_REG_T1, rs, ctx);

Will do it in next version.

Quoted 5 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);

ditto, but for or and mv

How would we combine or and mv?
also, we have a beq above and in one case both or and mv should happen, but in other case only mv should happen.

Thanks,
Puranjay

#7 lore raw

Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

In reply to Puranjay Mohan
Quoted 9 lines · Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.

rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY

Quoted 6 lines

rY = addr_space_cast(rX, 1, 0) has to be converted by JIT:

Here I explain using symbolic language what the JIT is supposed to do:
We have:
src = [src_upper32][src_lower32] // 64 bit src kernel pointer uvm = [uvm_upper32][uvm_lower32] // 64 bit user_vm_start

This is a bit misleading.
src_lower32 are always equal to uvm_lower32
and src_upper32 are either zero or uvm_upper32.

Hence most of the time llvm doesn't generate this insn, since it knows that upper 32 bit are uvm_upper32.

Quoted 28 lines

The JIT has to make the dst reg like following

        dst = [uvm_upper32][src_lower32] // if src_lower32 != 0
        dst = [00000000000][00000000000] // if src_lower32 == 0
---
 arch/riscv/net/bpf_jit.h        |  1 +
 arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++
 arch/riscv/net/bpf_jit_core.c   |  1 +
 3 files changed, 17 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
index 8a47da08dd9c..5fc374ed98ea 100644
@@ -82,6 +82,7 @@ struct rv_jit_context {
        unsigned long flags;
        int stack_size;
        u64 arena_vm_start;
+       u64 user_vm_start;
 };
 
 /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +0 -0
index f51b832eafb6..3c389e75cb96 100644

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1087

Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c +2 -0
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
        /* dst = src */
        case BPF_ALU | BPF_MOV | BPF_X:
        case BPF_ALU64 | BPF_MOV | BPF_X:
+               if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+                   insn->imm == 1U << 16) {

Let's add a generic helper like insn_is_zext(), call it insn_is_cast_user() ?
and use it in all JIT-s ?
I should have added it right away when I did x86 part. Sorry.

And a comment next to the helper that addr space cast 0->1 is for converting bpf arena pointers to user vma. Hence the name.

Same comments for arm64 JIT arena support.

pw-bot: cr

maintainer
#6 lore raw

Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction

In reply to Puranjay Mohan
Quoted 32 lines · Puranjay Mohan
Quoted 4 lines · Pu Lehui
Quoted 1 line · Puranjay Mohan

LLVM generates bpf_addr_space_cast instruction while translating

[snip]

Quoted 1 line
Quoted 13 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088

Quoted 12 lines
   /* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +3 -0
index f51b832eafb6..3c389e75cb96 100644
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
   	/* dst = src */
   	case BPF_ALU | BPF_MOV | BPF_X:
   	case BPF_ALU64 | BPF_MOV | BPF_X:
+		if (BPF_CLASS(insn->code) == BPF_ALU64 && insn->off == BPF_ADDR_SPACE_CAST &&
+		    insn->imm == 1U << 16) {
+			emit_mv(RV_REG_T1, rs, ctx); > +			emit_zextw(RV_REG_T1, RV_REG_T1, ctx);

combine mv and zextw will be better

Do you suggest doing:

emit_zextw(RV_REG_T1, rs, ctx);

Will do it in next version.

Quoted 5 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+			emit_imm(rd, (ctx->user_vm_start >> 32) << 32, ctx);
+			emit(rv_beq(RV_REG_T1, RV_REG_ZERO, 4), ctx);
+			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
+			emit_mv(rd, RV_REG_T1, ctx);

ditto, but for or and mv

Quoted 4 lines

How would we combine or and mv?
also, we have a beq above and in one case both or and mv should happen, but in other case only mv should happen.

Okay, another branch is that t1 is zero, but not rd.

Quoted 2 lines

Thanks,
Puranjay