riscv, bpf: Support per-CPU insn and inline bpf_get_smp_processor_id()
Subsystems (4)
Puranjay Mohan
reviewerv2 cover riscv, bpf: Support per-CPU insn and inline bpf_get_smp_processor_id()
Changes in v1->v2:
v1: https://lore.kernel.org/all/[email protected]/
- Use emit_addr() in place of emit_imm() in the first patch.
- Add the second patch to inline bpf_get_smp_processor_id()
The first patch add the support for the internal-only MOV instruction to resolve per-CPU addrs to absolute addresses.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).
The second patch inlines the calls bpf_get_smp_processor_id() in the JIT by emitting a load from the TP + offsetof(struct thread_info, cpu).
Benchmark using [1] on Qemu.
./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc
+---------------+------------------+------------------+--------------+ | Name | Before | After | % change | |---------------+------------------+------------------+--------------| | glob-arr-inc | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s | + 24.04% | | arr-inc | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s | + 23.56% | | hash-inc | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s | + 32.18% | +---------------+------------------+------------------+--------------+
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Puranjay Mohan (2):
riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs riscv, bpf: inline bpf_get_smp_processor_id()
arch/riscv/net/bpf_jit_comp64.c | 50 +++++++++++++++++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 ++++++++ kernel/bpf/verifier.c | 2 ++ 4 files changed, 64 insertions(+)
Signature · 1 line
2.40.1
Puranjay Mohan
reviewerPATCH v2 1/2 riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).
Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.
To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/s
After:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +24 -0
@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx); +#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu), +RV_REG_TP , ctx); +/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_addr (RV_REG_T2 , (u64)&__per_cpu_offset, extra_pass, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx); +#endif }if (imm ==1 ) {/* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void) {return true; } + +bool bpf_jit_supports_percpu_insn (void ) +{ +return true; +}
Signature · 1 line
2.40.1
Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Quoted 60 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intask_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +8
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx); +#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu), +RV_REG_TP , ctx);
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1109
Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10
-0+/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_addr (RV_REG_T2 , (u64)&__per_cpu_offset, extra_pass, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx);
is this the right level of code indentation?
Quoted 16 lines
arch/riscv/net/bpf_jit_comp64.c +6
-1+#endif }if (imm ==1 ) {/* Special mov32 for zext */ @@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void) {return true; } + +bool bpf_jit_supports_percpu_insn (void ) +{ +return true; +} --2.40.1
Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Quoted 49 lines · Andrii Nakryiko
Quoted 48 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intask_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h"
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0#define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx);
No biggie, but you did not fold this check into emit_mv().
Quoted 15 lines
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu), +RV_REG_TP , ctx);arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1109
Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10
-0+/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_addr (RV_REG_T2 , (u64)&__per_cpu_offset, extra_pass, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx);
Quoted 1 line
is this the right level of code indentation?
Looks wrong.
When the indent is fixed, feel free to add:
Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Quoted 78 lines · Björn Töpel
Quoted 49 lines · Andrii Nakryiko
Quoted 48 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intask_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0#define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx);No biggie, but you did not fold this check into
emit_mv().Quoted 15 lines
Quoted 14 lines
arch/riscv/net/bpf_jit_comp64.c +14
-0+#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu), +RV_REG_TP , ctx); +/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_addr (RV_REG_T2 , (u64)&__per_cpu_offset, extra_pass, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx);
Quoted 7 lines
Quoted 1 line
is this the right level of code indentation?
Looks wrong.
When the indent is fixed, feel free to add:
Acked-by Björn Töpel <[email protected]>
I fixed the indent and sent it as part of:
https://lore.kernel.org/all/[email protected]/
Also, for the emit_mv() thing, I wanted to verify if we are using that somewhere for zero-extension or something.
So, I thought I would send a separate patch for it.
Thanks,
Puranjay
Puranjay Mohan
reviewerPATCH v2 2/2 riscv, bpf: inline bpf_get_smp_processor_id()
In reply to Puranjay MohanInline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.
RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).
RISCV64 JIT output for `call bpf_get_smp_processor_id`
======================================================
Before After
-------- -------
auipc t1,0x848c ld a5,32(tp)
jalr 604(t1)
mv a5,a0Benchmark using [1] on Qemu.
./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc
+---------------+------------------+------------------+--------------+ | Name | Before | After | % change | |---------------+------------------+------------------+--------------| | glob-arr-inc | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s | + 24.04% | | arr-inc | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s | + 23.56% | | hash-inc | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s | + 32.18% | +---------------+------------------+------------------+--------------+
NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
--- arch/riscv/net/bpf_jit_comp64.c | 26 ++++++++++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 +++++++++++ kernel/bpf/verifier.c | 2 ++ 4 files changed, 40 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +26 -0
@@ -1493,6 +1493,22 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,bool fixed_addr;u64 addr; +/* Inline calls to bpf_get_smp_processor_id() +* +* RV_REG_TP holds the address of the current CPU's task_struct and thread_info is +* at offset 0 in task_struct. +* Load cpu from thread_info: +* Set R0 to ((struct thread_info *)(RV_REG_TP))->cpu +* +* This replicates the implementation of raw_smp_processor_id() on RISCV +*/ +if (insn->src_reg ==0 && insn->imm == BPF_FUNC_get_smp_processor_id) { +/* Load current CPU number in R0 */ +emit_ld (bpf_to_rv_reg (BPF_REG_0 , ctx), offsetof(struct thread_info , cpu), +RV_REG_TP , ctx); +break ; + } +mark_call (ctx); ret =bpf_jit_get_func_addr (ctx->prog, insn, extra_pass, &addr, &fixed_addr);
@@ -2062,3 +2078,13 @@ bool bpf_jit_supports_percpu_insn(void) {return true; } + +bool bpf_jit_inlines_helper_call (s32 imm) +{ +switch (imm) { +case BPF_FUNC_get_smp_processor_id: +return true; + } + +return false; +}
include/linux/filter.h +1 -0
@@ -993,6 +993,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);struct bpf_prog *bpf_int_jit_compile (struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void ); +bool bpf_jit_inlines_helper_call (s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );
kernel/bpf/core.c +11 -0
@@ -2941,6 +2941,17 @@ bool __weak bpf_jit_needs_zext(void)return false; } +/* Return true if the JIT inlines the call to the helper corresponding to +* the imm. +* +* The verifier will not patch the insn->imm for the call to the helper if +* this returns true. +*/ +bool __weak bpf_jit_inlines_helper_call (s32 imm) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {
kernel/bpf/verifier.c +2 -0
@@ -20013,6 +20013,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } +if (bpf_jit_inlines_helper_call (insn->imm)) + goto next_insn;if (insn->imm == BPF_FUNC_get_route_realm) prog->dst_needed =1 ;if (insn->imm == BPF_FUNC_get_prandom_u32)
Signature · 1 line
2.40.1
Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()
Quoted 30 lines · Puranjay Mohan
Inline the calls to
bpf_get_smp_processor_id()in the riscv bpf jit.RISCV saves the pointer to the CPU's
task_structin the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. Asthread_infois the first member oftask_struct, we can read the processor id from TP +offsetof(struct thread_info, cpu).RISCV64 JIT output for `call bpf_get_smp_processor_id` ====================================================== Before After -------- ------- auipc t1,0x848c ld a5,32(tp) jalr 604(t1) mv a5,a0Benchmark using [1] on Qemu.
./benchs/run_bench_trigger.shglob-arr-inc arr-inc hash-inc+---------------+------------------+------------------+--------------+ | Name | Before | After | % change | |---------------+------------------+------------------+--------------| | glob-arr-inc | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s | + 24.04% | | arr-inc | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s | + 23.56% | | hash-inc | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s | + 32.18% | +---------------+------------------+------------------+--------------+
Quoted 7 lines
NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>---
For non-riscv bits (& fwiw):
Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()
In reply to Puranjay MohanQuoted 8 lines · Puranjay Mohan
Inline the calls to
bpf_get_smp_processor_id()in the riscv bpf jit.RISCV saves the pointer to the CPU's
task_structin the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. Asthread_infois the first member oftask_struct, we can read the processor id from TP +offsetof(struct thread_info, cpu).
Quoted 10 lines
RISCV64 JIT output for `call bpf_get_smp_processor_id`
======================================================
Before After
-------- -------
auipc t1,0x848c ld a5,32(tp)
jalr 604(t1)
mv a5,a0Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling bpf_get_smp_processor_id for x86-64 shows this:
Dump of assembler code for function bpf_get_smp_processor_id:
0xffffffff810f91a0 <+0>: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0xffffffff810f91a5 <+5>: 65 8b 05 60 79 f3 7e mov
%gs:0x7ef37960(%rip),%eax # 0x30b0c <pcpu_hot+12>
0xffffffff810f91ac <+12>: 48 98 cltq 0xffffffff810f91ae <+14>: c3 ret
End of assembler dump.
We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").
Anyways, great work, a small nit below.
Quoted 55 lines
Benchmark using [1] on Qemu.
./benchs/run_bench_trigger.shglob-arr-inc arr-inc hash-inc+---------------+------------------+------------------+--------------+ | Name | Before | After | % change | |---------------+------------------+------------------+--------------| | glob-arr-inc | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s | + 24.04% | | arr-inc | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s | + 23.56% | | hash-inc | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s | + 32.18% | +---------------+------------------+------------------+--------------+NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 26 ++++++++++++++++++++++++++ include/linux/filter.h | 1 + kernel/bpf/core.c | 11 +++++++++++ kernel/bpf/verifier.c | 2 ++ 4 files changed, 40 insertions(+)arch/riscv/net/bpf_jit_comp64.c +16
-0@@ -1493,6 +1493,22 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,bool fixed_addr;u64 addr; +/* Inline calls to bpf_get_smp_processor_id() +* +* RV_REG_TP holds the address of the current CPU's task_struct and thread_info is +* at offset 0 in task_struct. +* Load cpu from thread_info: +* Set R0 to ((struct thread_info *)(RV_REG_TP))->cpu +* +* This replicates the implementation of raw_smp_processor_id() on RISCV +*/ +if (insn->src_reg ==0 && insn->imm == BPF_FUNC_get_smp_processor_id) { +/* Load current CPU number in R0 */ +emit_ld (bpf_to_rv_reg (BPF_REG_0 , ctx), offsetof(struct thread_info , cpu), +RV_REG_TP , ctx); +break ; + } +mark_call (ctx); ret =bpf_jit_get_func_addr (ctx->prog, insn, extra_pass, &addr, &fixed_addr);@@ -2062,3 +2078,13 @@ bool bpf_jit_supports_percpu_insn(void) {return true;
arch/riscv/net/bpf_jit_comp64.c bool bpf_jit_supports_percpu_insn(void) · line 2089
Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +9
-0} + +bool bpf_jit_inlines_helper_call (s32 imm) +{ +switch (imm) { +case BPF_FUNC_get_smp_processor_id: +return true; + } + +return false;
nit: why not
default:
return false;
to keep everything within the switch?
Quoted 40 lines
arch/riscv/net/bpf_jit_comp64.c +1
-0+}include/linux/filter.h +1
-0@@ -993,6 +993,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);struct bpf_prog *bpf_int_jit_compile (struct bpf_prog *prog);void bpf_jit_compile (struct bpf_prog *prog);bool bpf_jit_needs_zext (void ); +bool bpf_jit_inlines_helper_call (s32 imm);bool bpf_jit_supports_subprog_tailcalls (void );bool bpf_jit_supports_percpu_insn (void );bool bpf_jit_supports_kfunc_call (void );kernel/bpf/core.c +11
-0@@ -2941,6 +2941,17 @@ bool __weak bpf_jit_needs_zext(void)return false; } +/* Return true if the JIT inlines the call to the helper corresponding to +* the imm. +* +* The verifier will not patch the insn->imm for the call to the helper if +* this returns true. +*/ +bool __weak bpf_jit_inlines_helper_call (s32 imm) +{ +return false; +} +/* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */ bool __weak bpf_jit_supports_subprog_tailcalls (void ) {kernel/bpf/verifier.c +0
-0
kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 20017
Quoted 5 lines
kernel/bpf/verifier.c +2
-0@@ -20013,6 +20013,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env) goto next_insn; } +if (bpf_jit_inlines_helper_call (insn->imm)) + goto next_insn;
It's nice to be able to allow BPF JIT to do a higher-performance implementation. Let's add a short comment above to mention that this is bypassing normal inlining because BPF JIT will do it better (I know you have this description for the function definition, but a short remark here would be helpful).
And please add an empty line after this check to logically separate it from the rest of helper inlining logic in verifier, thanks!
pw-bot: cr
Quoted 6 lines
kernel/bpf/verifier.c +0
-1if (insn->imm == BPF_FUNC_get_route_realm) prog->dst_needed =1 ;if (insn->imm == BPF_FUNC_get_prandom_u32) --2.40.1
Puranjay Mohan
reviewerRe: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()
Quoted 29 lines · Andrii Nakryiko
Quoted 8 lines · Puranjay Mohan
Inline the calls to
bpf_get_smp_processor_id()in the riscv bpf jit.RISCV saves the pointer to the CPU's
task_structin the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. Asthread_infois the first member oftask_struct, we can read the processor id from TP +offsetof(struct thread_info, cpu).Quoted 10 lines
RISCV64 JIT output for `call bpf_get_smp_processor_id` ====================================================== Before After -------- ------- auipc t1,0x848c ld a5,32(tp) jalr 604(t1) mv a5,a0Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling
bpf_get_smp_processor_idfor x86-64 shows this:Dump of assembler code for function
bpf_get_smp_processor_id:0xffffffff810f91a0 <+0>: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 0xffffffff810f91a5 <+5>: 65 8b 05 60 79 f3 7e mov %gs:0x7ef37960(%rip),%eax # 0x30b0c <pcpu_hot+12> 0xffffffff810f91ac <+12>: 48 98 cltq 0xffffffff810f91ae <+14>: c3 ret
Quoted 4 lines
End of assembler dump.
We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").
I feel in x86-64's case JIT can not do a (much) better job compared to the current approach in the verifier.
On RISC-V and ARM64, JIT was able to do it better because both of these architectures save a pointer to the task struct in a special CPU register. As x86-64 doesn't have enough extra registers, it uses a percpu variable to store task struct, thread_info, and the cpu number.
P.S. - While doing this for BPF, I realized that ARM64 kernel code is also not optimal as it is using the percpu variable and is not reading the CPU register directly. So, I sent a patch[1] to fix it in the kernel and get rid of the per-cpu variable in ARM64.
[1] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Anyways, great work, a small nit below.
Acked-by Andrii Nakryiko <[email protected]>
Thanks,
Puranjay
Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()
Quoted 37 lines · Puranjay Mohan
Quoted 29 lines · Andrii Nakryiko
Quoted 8 lines · Puranjay Mohan
Inline the calls to
bpf_get_smp_processor_id()in the riscv bpf jit.RISCV saves the pointer to the CPU's
task_structin the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. Asthread_infois the first member oftask_struct, we can read the processor id from TP +offsetof(struct thread_info, cpu).Quoted 10 lines
RISCV64 JIT output for `call bpf_get_smp_processor_id` ====================================================== Before After -------- ------- auipc t1,0x848c ld a5,32(tp) jalr 604(t1) mv a5,a0Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling
bpf_get_smp_processor_idfor x86-64 shows this:Dump of assembler code for function
bpf_get_smp_processor_id:0xffffffff810f91a0 <+0>: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1) 0xffffffff810f91a5 <+5>: 65 8b 05 60 79 f3 7e mov %gs:0x7ef37960(%rip),%eax # 0x30b0c <pcpu_hot+12> 0xffffffff810f91ac <+12>: 48 98 cltq 0xffffffff810f91ae <+14>: c3 retQuoted 4 lines
End of assembler dump.
We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").
Quoted 2 lines
I feel in x86-64's case JIT can not do a (much) better job compared to the current approach in the verifier.
This direct memory read (using gs segment) ought to be a bit faster than calculating offset and then doing memory dereference, but yes, the difference won't be as big as you got with RISC-V and ARM64. Ok, never mind, we can always benchmark and add that later, no big deal.
Quoted 21 lines
On RISC-V and ARM64, JIT was able to do it better because both of these architectures save a pointer to the task struct in a special CPU register. As x86-64 doesn't have enough extra registers, it uses a percpu variable to store task struct,
thread_info, and the cpu number.
P.S. - While doing this for BPF, I realized that ARM64 kernel code is also not optimal as it is using the percpu variable and is not reading the CPU register directly. So, I sent a patch[1] to fix it in the kernel and get rid of the per-cpu variable in ARM64.[1] https://lore.kernel.org/all/[email protected]/
Quoted 3 lines
Anyways, great work, a small nit below.
Acked-by Andrii Nakryiko <[email protected]>Thanks,
Puranjay
Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()
In reply to Puranjay MohanQuoted 29 lines · Puranjay Mohan
Inline the calls to
bpf_get_smp_processor_id()in the riscv bpf jit.RISCV saves the pointer to the CPU's
task_structin the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. Asthread_infois the first member oftask_struct, we can read the processor id from TP +offsetof(struct thread_info, cpu).RISCV64 JIT output for `call bpf_get_smp_processor_id` ====================================================== Before After -------- ------- auipc t1,0x848c ld a5,32(tp) jalr 604(t1) mv a5,a0Benchmark using [1] on Qemu.
./benchs/run_bench_trigger.shglob-arr-inc arr-inc hash-inc+---------------+------------------+------------------+--------------+ | Name | Before | After | % change | |---------------+------------------+------------------+--------------| | glob-arr-inc | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s | + 24.04% | | arr-inc | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s | + 23.56% | | hash-inc | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s | + 32.18% | +---------------+------------------+------------------+--------------+
Quoted 6 lines
NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>