bpf,riscv: Add support for BPF Arena
Subsystems (4)
v3 cover bpf,riscv: Add support for BPF Arena
Changes in V3
V2: https://lore.kernel.org/bpf/[email protected]/
- Use REG_DONT_CLEAR_MARKER in place of DONT_CLEAR as the name.
- Use RV_REG_ZERO for the above as it is guaranteed to be never used in as dst/src register in BPF LDX/STX/ST instructions.
- Use #define RV_REG_ARENA for RV_REG_S7
Changes in V2
V1: https://lore.kernel.org/bpf/[email protected]/
- Used S7 in place of S11 for storing the kernel_vm_start
- Used 17 in place of 16 for DONT_CLEAR marker
- Remove an unused variable
- Removed some misleading information from the commit message.
This series adds the support for PROBE_MEM32 and bpf_addr_space_cast instructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.
All arena related selftests are passing:
root@rv-tester:~/bpf# uname -p riscv64 root@rv-tester:~/bpf# ./test_progs -a "*arena*" #3/1 arena_htab/arena_htab_llvm:OK #3/2 arena_htab/arena_htab_asm:OK #3 arena_htab:OK #4/1 arena_list/arena_list_1:OK #4/2 arena_list/arena_list_1000:OK #4 arena_list:OK #434/1 verifier_arena/basic_alloc1:OK #434/2 verifier_arena/basic_alloc2:OK #434/3 verifier_arena/basic_alloc3:OK #434/4 verifier_arena/iter_maps1:OK #434/5 verifier_arena/iter_maps2:OK #434/6 verifier_arena/iter_maps3:OK #434 verifier_arena:OK Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED
This feature needs the following two fixes in the bpf/bpf.git to work properly:
f7f5d1808b1b6 ("bpf: verifier: fix addr_space_cast from as(1) to as(0)") 443574b033876 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")
There is a lot of code repetition for LDX, STX, and ST. I will be sending a follow-up patch to refactor these.
Puranjay Mohan (2):
bpf,riscv: Implement PROBE_MEM32 pseudo instructions bpf,riscv: Implement bpf_addr_space_cast instruction
arch/riscv/net/bpf_jit.h | 2 + arch/riscv/net/bpf_jit_comp64.c | 205 +++++++++++++++++++++++++++++++- arch/riscv/net/bpf_jit_core.c | 2 + 3 files changed, 207 insertions(+), 2 deletions(-)
Signature · 1 line
2.40.1
PATCH v3 1/2 bpf,riscv: Implement PROBE_MEM32 pseudo instructions
Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences:
- PROBE_MEM32 supports store.
- PROBE_MEM32 relies on the verifier to clear upper 32-bit of the src/dst register
- PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in S7 in the prologue). Due to bpf_arena constructions such S7 + reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time.
- S11 is a free callee-saved register, so it is used to store kern_vm_start
- PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed.
To support these on riscv, we do tmp = S7 + src/dst reg and then use tmp2 as the new src/dst register. This allows us to reuse most of the code for normal [LDX | STX | ST].
--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 191 +++++++++++++++++++++++++++++++- arch/riscv/net/bpf_jit_core.c | 1 + 3 files changed, 191 insertions(+), 2 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
@@ -81,6 +81,7 @@ struct rv_jit_context {int nexentries;unsigned long flags;int stack_size; +u64 arena_vm_start; };/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +189 -2
263 lines not on this page · raw message
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -80,6 +80,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto skip_init_ctx; } + ctx->arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); ctx->prog = prog; ctx->offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (!ctx->offset) {
Signature · 1 line
2.40.1
Re: [PATCH bpf-next v3 1/2] bpf,riscv: Implement PROBE_MEM32 pseudo instructions
Quoted 1 line · Puranjay Mohan
Add support for [LDX | STX | ST],
PROBE_MEM32, [B | H | W | DW]
[SNIP]
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1550
Quoted 6 lines
if (
WARN_ON_ONCE(ctx->nexentries >= ctx->prog->aux->num_exentries))arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -1539,6 +1547,11 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,case BPF_LDX |BPF_PROBE_MEMSX |BPF_B :case BPF_LDX |BPF_PROBE_MEMSX |BPF_H :case BPF_LDX |BPF_PROBE_MEMSX |BPF_W : +/* LDX | PROBE_MEM32: dst = *(unsigned size *)(src + S7 + off)*/
still some nits. please align, change S7 here to reg_arena.
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+case BPF_LDX |BPF_PROBE_MEM32 |BPF_B : +case BPF_LDX |BPF_PROBE_MEM32 |BPF_H : +case BPF_LDX |BPF_PROBE_MEM32 |BPF_W : +case BPF_LDX |BPF_PROBE_MEM32 |BPF_DW :
[SNIP]
Quoted 6 lines
arch/riscv/net/bpf_jit_comp64.c +6
-0+ +switch (BPF_SIZE (code)) { +case BPF_B : +if (is_12b_int (off)) { + insns_start = ctx->ninsns; +emit (rv_sb (rd, off,RV_REG_T1 ), ctx);
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1730
Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10
-0+ insn_len = ctx->ninsns - insns_start; +break ; + } + +emit_imm (RV_REG_T2 , off, ctx); +emit_add (RV_REG_T2 ,RV_REG_T2 , rd, ctx); + insns_start = ctx->ninsns; +emit (rv_sb (RV_REG_T2 ,0 ,RV_REG_T1 ), ctx); + insn_len = ctx->ninsns - insns_start; +
redundant blank.
Review
Quoted 2 lines
(quoted diff) +2
-0+ break; +
ditto.
Others, looks good to me.
PATCH v3 2/2 bpf,riscv: Implement bpf_addr_space_cast instruction
In reply to Puranjay MohanLLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.
rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY
rY = addr_space_cast(rX, 1, 0) has to be converted by JIT.
--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 14 ++++++++++++++ arch/riscv/net/bpf_jit_core.c | 1 + 3 files changed, 16 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
@@ -82,6 +82,7 @@ struct rv_jit_context {unsigned long flags;int stack_size;u64 arena_vm_start; +u64 user_vm_start; };/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +14 -0
@@ -1081,6 +1081,15 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (insn_is_cast_user (insn)) { +emit_mv (RV_REG_T1 , rs, ctx); +emit_zextw (RV_REG_T1 ,RV_REG_T1 , ctx); +emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx); +break ; + }if (imm ==1 ) {/* Special mov32 for zext */ emit_zextw (rd, rd, ctx);
@@ -2026,3 +2035,8 @@ bool bpf_jit_supports_ptr_xchg(void) {return true; } + +bool bpf_jit_supports_arena (void ) +{ +return true; +}
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -81,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) } ctx->arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); + ctx->user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx->prog = prog; ctx->offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (!ctx->offset) {
Signature · 1 line
2.40.1
Re: [PATCH bpf-next v3 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction
Quoted 63 lines · Puranjay Mohan
LLVM generates
bpf_addr_space_castinstruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved asbpf_arenaaddress space.rY =
addr_space_cast(rX, 0, 1)is processed by the verifier and converted to normal 32-bit move: wX = wYrY =
addr_space_cast(rX, 1, 0)has to be converted by JIT.Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 14 ++++++++++++++ arch/riscv/net/bpf_jit_core.c | 1 + 3 files changed, 16 insertions(+)arch/riscv/net/bpf_jit.h +1
-0@@ -82,6 +82,7 @@ struct rv_jit_context {unsigned long flags;int stack_size;u64 arena_vm_start; +u64 user_vm_start; };/* Convert from ninsns to bytes. */ arch/riscv/net/bpf_jit_comp64.c +14
-0@@ -1081,6 +1081,15 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (insn_is_cast_user (insn)) { +emit_mv (RV_REG_T1 , rs, ctx); +emit_zextw (RV_REG_T1 ,RV_REG_T1 , ctx); +emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx); +break ; + }if (imm ==1 ) {/* Special mov32 for zext */ emit_zextw (rd, rd, ctx);@@ -2026,3 +2035,8 @@ bool bpf_jit_supports_ptr_xchg(void) {return true; } + +bool bpf_jit_supports_arena (void ) +{ +return true; +}arch/riscv/net/bpf_jit_core.c +0
-0
arch/riscv/net/bpf_jit_core.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 87
Quoted 7 lines
arch/riscv/net/bpf_jit_core.c +1
-0@@ -81,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) } ctx->arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); + ctx->user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx->prog = prog; ctx->offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (!ctx->offset) {
Quoted 39 lines · Puranjay Mohan
Changes in V3
V2: https://lore.kernel.org/bpf/[email protected]/
- UseREG_DONT_CLEAR_MARKERin place ofDONT_CLEARas the name.
- UseRV_REG_ZEROfor the above as it is guaranteed to be never used in as dst/src register in BPF LDX/STX/ST instructions.
- Use #defineRV_REG_ARENAforRV_REG_S7Changes in V2
V1: https://lore.kernel.org/bpf/[email protected]/
- Used S7 in place of S11 for storing thekernel_vm_start
- Used 17 in place of 16 forDONT_CLEARmarker
- Remove an unused variable
- Removed some misleading information from the commit message.This series adds the support for
PROBE_MEM32andbpf_addr_space_castinstructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.All arena related selftests are passing:
root@rv-tester:~/bpf# uname -p riscv64 root@rv-tester:~/bpf# ./test_progs -a "*arena*" #3/1 arena_htab/arena_htab_llvm:OK #3/2 arena_htab/arena_htab_asm:OK #3 arena_htab:OK #4/1 arena_list/arena_list_1:OK #4/2 arena_list/arena_list_1000:OK #4 arena_list:OK #434/1 verifier_arena/basic_alloc1:OK #434/2 verifier_arena/basic_alloc2:OK #434/3 verifier_arena/basic_alloc3:OK #434/4 verifier_arena/iter_maps1:OK #434/5 verifier_arena/iter_maps2:OK #434/6 verifier_arena/iter_maps3:OK #434 verifier_arena:OK Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED
Quoted 8 lines
This feature needs the following two fixes in the bpf/bpf.git to work properly:
f7f5d1808b1b6("bpf: verifier: fixaddr_space_castfrom as(1) to as(0)")443574b033876("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")There is a lot of code repetition for LDX, STX, and ST. I will be sending a follow-up patch to refactor these.
Thank you, and please address Lehui's remarks as well.
For the series: