bpf,riscv: Add support for BPF Arena
Subsystems (4)
cover bpf,riscv: Add support for BPF Arena
This series adds the support for PROBE_MEM32 and bpf_addr_space_cast instructions to the RISCV BPF JIT. These two instructions allow the enablement of BPF Arena.
All arena related selftests are passing:
root@rv-tester:~/bpf# uname -p riscv64 root@rv-tester:~/bpf# ./test_progs -a "*arena*" #3/1 arena_htab/arena_htab_llvm:OK #3/2 arena_htab/arena_htab_asm:OK #3 arena_htab:OK #4/1 arena_list/arena_list_1:OK #4/2 arena_list/arena_list_1000:OK #4 arena_list:OK #434/1 verifier_arena/basic_alloc1:OK #434/2 verifier_arena/basic_alloc2:OK #434/3 verifier_arena/basic_alloc3:OK #434/4 verifier_arena/iter_maps1:OK #434/5 verifier_arena/iter_maps2:OK #434/6 verifier_arena/iter_maps3:OK #434 verifier_arena:OK Summary: 3/10 PASSED, 0 SKIPPED, 0 FAILED
This needs the following commit in bpf/bpf.git to work correctly:
https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git/commit/?id=f7f5d1808b1b66935a24dd796dd1a0612ca9c147
Puranjay Mohan (2):
bpf,riscv: Implement PROBE_MEM32 pseudo instructions bpf,riscv: Implement bpf_addr_space_cast instruction
arch/riscv/net/bpf_jit.h | 2 + arch/riscv/net/bpf_jit_comp64.c | 208 +++++++++++++++++++++++++++++++- arch/riscv/net/bpf_jit_core.c | 3 + 3 files changed, 210 insertions(+), 3 deletions(-)
Signature · 1 line
2.40.1
PATCH 1/2 bpf,riscv: Implement PROBE_MEM32 pseudo instructions
Add support for [LDX | STX | ST], PROBE_MEM32, [B | H | W | DW] instructions. They are similar to PROBE_MEM instructions with the following differences:
- PROBE_MEM32 supports store.
- PROBE_MEM32 relies on the verifier to clear upper 32-bit of the src/dst register
- PROBE_MEM32 adds 64-bit kern_vm_start address (which is stored in S11 in the prologue). Due to bpf_arena constructions such S11 + reg + off16 access is guaranteed to be within arena virtual range, so no address check at run-time.
- S11 is a free callee-saved register, so it is used to store kern_vm_start
- PROBE_MEM32 allows STX and ST. If they fault the store is a nop. When LDX faults the destination register is zeroed.
To support these on riscv, we do tmp = S11 + src/dst reg and then use tmp2 as the new src/dst register. This allows us to reuse most of the code for normal [LDX | STX | ST].
--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 193 +++++++++++++++++++++++++++++++- arch/riscv/net/bpf_jit_core.c | 2 + 3 files changed, 193 insertions(+), 3 deletions(-)
arch/riscv/net/bpf_jit.h +1 -0
@@ -81,6 +81,7 @@ struct rv_jit_context {int nexentries;unsigned long flags;int stack_size; +u64 arena_vm_start; };/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +190 -3
271 lines not on this page · raw message
arch/riscv/net/bpf_jit_core.c +2 -0
@@ -50,6 +50,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)int pass =0 , prev_ninsns =0 , i;struct rv_jit_data *jit_data;struct rv_jit_context *ctx; +u64 arena_vm_start;if (!prog->jit_requested)return orig_prog;
@@ -80,6 +81,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) goto skip_init_ctx; } + ctx->arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); ctx->prog = prog; ctx->offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (!ctx->offset) {
Signature · 1 line
2.40.1
Pu Lehui
maintainerRe: [PATCH bpf-next 1/2] bpf,riscv: Implement PROBE_MEM32 pseudo instructions
Quoted 1 line · Puranjay Mohan
Add support for [LDX | STX | ST],
PROBE_MEM32, [B | H | W | DW]
[snip]
Quoted 4 lines
#define
BPF_FIXUP_OFFSET_MASKGENMASK(26, 0)
#defineBPF_FIXUP_REG_MASKGENMASK(31, 27)
+#defineDONT_CLEAR16 /*RV_REG_A6unused in BPF */
This is a bit misleading. RV_REG_A6 is actually used in riscv64. Maybe "RV_REG_A6 unused in pt_regmap" or change to other register will be better.
Quoted 2 lines
bool ex_handler_bpf(const struct
exception_table_entry*ex,
[snip]
arch/riscv/net/bpf_jit_comp64.c void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog) · line 1980
Quoted 11 lines
stack_adjust = round_up(stack_adjust, 16); stack_adjust += bpf_stack_adjust;arch/riscv/net/bpf_jit_comp64.c +4
-0@@ -1794,6 +1974,10 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)emit_sd (RV_REG_SP , store_offset,RV_REG_S6 , ctx); store_offset -=8 ; } +if (ctx->arena_vm_start) { +emit_sd (RV_REG_SP , store_offset,RV_REG_S11 , ctx); + store_offset -=8 ; + }
I think it's fine to use s7 and keep the original dynamic stack code style.
Quoted 18 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0emit_addi (RV_REG_FP ,RV_REG_SP , stack_adjust, ctx);@@ -1807,6 +1991,9 @@ void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)emit_mv (RV_REG_TCC_SAVED ,RV_REG_TCC , ctx); ctx->stack_size = stack_adjust; + +if (ctx->arena_vm_start) +emit_imm (RV_REG_S11 , ctx->arena_vm_start, ctx); }void bpf_jit_build_epilogue (struct rv_jit_context *ctx)arch/riscv/net/bpf_jit_core.c +0
-0
arch/riscv/net/bpf_jit_core.c struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) · line 53
Quoted 4 lines
arch/riscv/net/bpf_jit_core.c +1
-0@@ -50,6 +50,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)int pass =0 , prev_ninsns =0 , i;struct rv_jit_data *jit_data;struct rv_jit_context *ctx; +u64 arena_vm_start;
unused variable
PATCH 2/2 bpf,riscv: Implement bpf_addr_space_cast instruction
In reply to Puranjay MohanLLVM generates bpf_addr_space_cast instruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved as bpf_arena address space.
rY = addr_space_cast(rX, 0, 1) is processed by the verifier and converted to normal 32-bit move: wX = wY
rY = addr_space_cast(rX, 1, 0) has to be converted by JIT:
Here I explain using symbolic language what the JIT is supposed to do:
We have:
src = [src_upper32][src_lower32] // 64 bit src kernel pointer uvm = [uvm_upper32][uvm_lower32] // 64 bit user_vm_start
The JIT has to make the dst reg like following
dst = [uvm_upper32][src_lower32] // if src_lower32 != 0 dst = [00000000000][00000000000] // if src_lower32 == 0
--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++ arch/riscv/net/bpf_jit_core.c | 1 + 3 files changed, 17 insertions(+)
arch/riscv/net/bpf_jit.h +1 -0
@@ -82,6 +82,7 @@ struct rv_jit_context {unsigned long flags;int stack_size;u64 arena_vm_start; +u64 user_vm_start; };/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +15 -0
@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (BPF_CLASS (insn->code) ==BPF_ALU64 && insn->off ==BPF_ADDR_SPACE_CAST && + insn->imm ==1U <<16 ) { +emit_mv (RV_REG_T1 , rs, ctx); +emit_zextw (RV_REG_T1 ,RV_REG_T1 , ctx); +emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx); +break ; + }if (imm ==1 ) {/* Special mov32 for zext */ emit_zextw (rd, rd, ctx);
@@ -2010,3 +2020,8 @@ bool bpf_jit_supports_ptr_xchg(void) {return true; } + +bool bpf_jit_supports_arena (void ) +{ +return true; +}
arch/riscv/net/bpf_jit_core.c +1 -0
@@ -82,6 +82,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) } ctx->arena_vm_start =bpf_arena_get_kern_vm_start (prog->aux->arena); + ctx->user_vm_start =bpf_arena_get_user_vm_start (prog->aux->arena); ctx->prog = prog; ctx->offset =kcalloc (prog->len,sizeof (int ),GFP_KERNEL );if (!ctx->offset) {
Signature · 1 line
2.40.1
Pu Lehui
maintainerRe: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction
Quoted 1 line · Puranjay Mohan
LLVM generates
bpf_addr_space_castinstruction while translating
[snip]
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088
Quoted 13 lines
/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +3
-0@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (BPF_CLASS (insn->code) ==BPF_ALU64 && insn->off ==BPF_ADDR_SPACE_CAST && + insn->imm ==1U <<16 ) { + emit_mv(RV_REG_T1, rs, ctx); > + emit_zextw(RV_REG_T1, RV_REG_T1, ctx);
combine mv and zextw will be better
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx);
ditto, but for or and mv
Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction
Quoted 4 lines · Pu Lehui
Quoted 1 line · Puranjay Mohan
LLVM generates
bpf_addr_space_castinstruction while translating[snip]
Quoted 1 line
Quoted 13 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088
Quoted 12 lines
/* Convert from ninsns to bytes. */
arch/riscv/net/bpf_jit_comp64.c +3
-0@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (BPF_CLASS (insn->code) ==BPF_ALU64 && insn->off ==BPF_ADDR_SPACE_CAST && + insn->imm ==1U <<16 ) { + emit_mv(RV_REG_T1, rs, ctx); > + emit_zextw(RV_REG_T1, RV_REG_T1, ctx);combine mv and zextw will be better
Do you suggest doing:
emit_zextw(RV_REG_T1, rs, ctx);
Will do it in next version.
Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx);ditto, but for or and mv
How would we combine or and mv?
also, we have a beq above and in one case both or and mv should happen, but in other case only mv should happen.
Thanks,
Puranjay
Pu Lehui
maintainerRe: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction
Quoted 32 lines · Puranjay Mohan
Quoted 4 lines · Pu Lehui
Quoted 1 line · Puranjay Mohan
LLVM generates
bpf_addr_space_castinstruction while translating[snip]
Quoted 1 line
Quoted 13 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1088
Quoted 12 lines
/* Convert from ninsns to bytes. */arch/riscv/net/bpf_jit_comp64.c +3
-0@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (BPF_CLASS (insn->code) ==BPF_ALU64 && insn->off ==BPF_ADDR_SPACE_CAST && + insn->imm ==1U <<16 ) { + emit_mv(RV_REG_T1, rs, ctx); > + emit_zextw(RV_REG_T1, RV_REG_T1, ctx);combine mv and zextw will be better
Do you suggest doing:
emit_zextw(RV_REG_T1, rs, ctx);Will do it in next version.
Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1093
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+emit_imm (rd, (ctx->user_vm_start >>32 ) <<32 , ctx); +emit (rv_beq (RV_REG_T1 ,RV_REG_ZERO ,4 ), ctx); +emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx); +emit_mv (rd,RV_REG_T1 , ctx);ditto, but for or and mv
Quoted 4 lines
How would we combine or and mv?
also, we have a beq above and in one case both or and mv should happen, but in other case only mv should happen.
Okay, another branch is that t1 is zero, but not rd.
Quoted 2 lines
Thanks,
Puranjay
Re: [PATCH bpf-next 2/2] bpf,riscv: Implement bpf_addr_space_cast instruction
In reply to Puranjay MohanQuoted 9 lines · Puranjay Mohan
LLVM generates
bpf_addr_space_castinstruction while translating pointers between native (zero) address space and __attribute__((address_space(N))). The addr_space=0 is reserved asbpf_arenaaddress space.rY =
addr_space_cast(rX, 0, 1)is processed by the verifier and converted to normal 32-bit move: wX = wY
Quoted 6 lines
rY =
addr_space_cast(rX, 1, 0)has to be converted by JIT:Here I explain using symbolic language what the JIT is supposed to do:
We have:
src = [src_upper32][src_lower32] // 64 bit src kernel pointer uvm = [uvm_upper32][uvm_lower32] // 64 bituser_vm_start
This is a bit misleading.src_lower32 are always equal to uvm_lower32
and src_upper32 are either zero or uvm_upper32.
Hence most of the time llvm doesn't generate this insn, since it knows that upper 32 bit are uvm_upper32.
Quoted 28 lines
The JIT has to make the dst reg like following
dst = [uvm_upper32][src_lower32] // if src_lower32 != 0 dst = [00000000000][00000000000] // if src_lower32 == 0Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit.h | 1 + arch/riscv/net/bpf_jit_comp64.c | 15 +++++++++++++++ arch/riscv/net/bpf_jit_core.c | 1 + 3 files changed, 17 insertions(+)arch/riscv/net/bpf_jit.h +1
-0@@ -82,6 +82,7 @@ struct rv_jit_context {unsigned long flags;int stack_size;u64 arena_vm_start; +u64 user_vm_start; };/* Convert from ninsns to bytes. */ arch/riscv/net/bpf_jit_comp64.c +0
-0
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1087
Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c +2
-0@@ -1083,6 +1083,16 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,/* dst = src */ case BPF_ALU |BPF_MOV |BPF_X :case BPF_ALU64 |BPF_MOV |BPF_X : +if (BPF_CLASS (insn->code) ==BPF_ALU64 && insn->off ==BPF_ADDR_SPACE_CAST && + insn->imm ==1U <<16 ) {
Let's add a generic helper like insn_is_zext(), call it insn_is_cast_user() ?
and use it in all JIT-s ?
I should have added it right away when I did x86 part. Sorry.
And a comment next to the helper that addr space cast 0->1 is for converting bpf arena pointers to user vma. Hence the name.
Same comments for arm64 JIT arena support.
pw-bot: cr