riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Subsystems (3)
PATCH 1/1 riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).
Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.
To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/s
After:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +24 -0
@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx); +#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu),RV_REG_TP , + ctx); +/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_imm (RV_REG_T2 , (u64)&__per_cpu_offset, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx); +#endif }if (imm ==1 ) {/* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void) {return true; } + +bool bpf_jit_supports_percpu_insn (void ) +{ +return true; +}
Signature · 1 line
2.40.1
Björn Töpel
maintainerQuoted 10 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intast_structis structthread_info, and we can get the cpu number by
^
k ;-)Quoted 5 lines
reading from the TP register +
offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.
Just to clarify for readers; BPF programs are run with migrate disable, which means that on RT we can be preempted, which means that per-cpu operations are trickier (disabling interrupts/preemption).
However, this BPF instruction is about calculating the per-cpu address, so the look up can be inlined.
It's not a per-cpu *operation*.
Quoted 37 lines
To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +2
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) {
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1103
Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10
-0+if (rd != rs) +emit_mv (rd, rs, ctx); +#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu),RV_REG_TP , + ctx); +/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_imm (RV_REG_T2 , (u64)&__per_cpu_offset, ctx);
Did you try using emit_addr() here? I'd guess that'll be fewer instructions, no?
Björn
Quoted 8 lines · Björn Töpel
Quoted 6 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
Quoted 6 lines
Quoted 4 lines
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intast_structis structthread_info, and we can get the cpu number by^ k ;-)
I need to start using some kind of spell-check in vim :D.
Quoted 6 lines
Quoted 5 lines
reading from the TP register +
offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.
Quoted 8 lines
Just to clarify for readers; BPF programs are run with migrate disable, which means that on RT we can be preempted, which means that per-cpu operations are trickier (disabling interrupts/preemption).
However, this BPF instruction is about calculating the per-cpu address, so the look up can be inlined.
It's not a per-cpu *operation*.
Will add this information to the commit message.
Quoted 42 lines
Quoted 42 lines
To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +7
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx); +#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ + emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP,
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1103
Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c +5
-0+ ctx); +/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_imm (RV_REG_T2 , (u64)&__per_cpu_offset, ctx);Did you try using
emit_addr()here? I'd guess that'll be fewer instructions, no?
Yes, I should have used that, the address would always be in the range of auipc+addi right? I will try and see.
Thanks,
Puranjay
Quoted 21 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intast_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
Quoted 8 lines
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sAfter:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/s
TBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?
Quoted 21 lines
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" #define RV_FENTRY_NINSNS 2
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095
Quoted 6 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx);
Is this an unconditional move instruction? in x86-64, EMIT_mov checks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1099
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4
-0+#ifdef CONFIG_SMP +/* Load current CPU number in T1 */ +emit_ld (RV_REG_T1 , offsetof(struct thread_info , cpu),RV_REG_TP , + ctx);
nit: maybe keep this on the same line?
Quoted 26 lines
arch/riscv/net/bpf_jit_comp64.c +16
-1+/* << 3 because offsets are 8 bytes */ +emit_slli (RV_REG_T1 ,RV_REG_T1 ,3 , ctx); +/* Load address of __per_cpu_offset array in T2 */ +emit_imm (RV_REG_T2 , (u64)&__per_cpu_offset, ctx); +/* Add offset of current CPU to __per_cpu_offset */ +emit_add (RV_REG_T1 ,RV_REG_T2 ,RV_REG_T1 , ctx); +/* Load __per_cpu_offset[cpu] in T1 */ +emit_ld (RV_REG_T1 ,0 ,RV_REG_T1 , ctx); +/* Add the offset to Rd */ +emit_add (rd, rd,RV_REG_T1 , ctx); +#endif }if (imm ==1 ) {/* Special mov32 for zext */ @@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void) {return true; } + +bool bpf_jit_supports_percpu_insn (void ) +{ +return true; +} --2.40.1
Björn Töpel
maintainerQuoted 26 lines · Andrii Nakryiko
Quoted 25 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intast_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/s
Quoted 7 lines
Quoted 4 lines
After:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/sTBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?
I just ran it on a "VisionFive2" SBC:
BEFORE
======
glob-arr-inc : 11.586 ± 0.021M/s arr-inc : 10.892 ± 0.005M/s hash-inc : 1.517 ± 0.001M/s
AFTER
=====
glob-arr-inc : 11.893 ± 0.017M/s (+2.6%) arr-inc : 11.630 ± 0.020M/s (+6.8%) hash-inc : 1.543 ± 0.002M/s (+1.7%)
(It's early, and the coffee haven't kicked in, so I hope the calculations are correct...)
Quoted 19 lines
Quoted 19 lines
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h"
Quoted 12 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0#define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx);Is this an unconditional move instruction? in x86-64,
EMIT_movchecks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?
Yeah, it is. Folding the check into the emit sounds like a good idea.
Björn
Quoted 45 lines · Björn Töpel
Quoted 26 lines · Andrii Nakryiko
Quoted 25 lines · Puranjay Mohan
Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the
task_structin TP register. The first element intast_structis structthread_info, and we can get the cpu number by reading from the TP register +offsetof(struct thread_info, cpu).Once we have the cpu number in a register we read the offset for that cpu from address:
&__per_cpu_offset+cpu_number<< 3. Then we add this offset to the destination register.To measure the improvement from this change, the benchmark in [1] was used on Qemu:
Before:
glob-arr-inc : 1.127 ± 0.013M/s arr-inc : 1.121 ± 0.004M/s hash-inc : 0.681 ± 0.052M/sQuoted 7 lines
Quoted 4 lines
After:
glob-arr-inc : 1.138 ± 0.011M/s arr-inc : 1.366 ± 0.006M/s hash-inc : 0.676 ± 0.001M/sTBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?
I just ran it on a "VisionFive2" SBC:
BEFORE
======glob-arr-inc : 11.586 ± 0.021M/s arr-inc : 10.892 ± 0.005M/s hash-inc : 1.517 ± 0.001M/s
Quoted 6 lines
AFTER
=====glob-arr-inc : 11.893 ± 0.017M/s (+2.6%) arr-inc : 11.630 ± 0.020M/s (+6.8%) hash-inc : 1.543 ± 0.002M/s (+1.7%)
Nice, looks pretty reasonable (and especially if bpf_smp_get_current_id() gets inlined as well, the numbers should be even better)
Quoted 35 lines
(It's early, and the coffee haven't kicked in, so I hope the calculations are correct...)
Quoted 19 lines
Quoted 19 lines
[1] https://github.com/anakryiko/linux/commit/8dec900975ef
Signed-off-by Puranjay Mohan <[email protected]>--- arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)arch/riscv/net/bpf_jit_comp64.c +1
-0@@ -12,6 +12,7 @@#include <linux/stop_machine.h> #include <asm/patch.h> #include <asm/cfi.h> +#include <asm/percpu.h> #include "bpf_jit.h" Quoted 12 lines
arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095
Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3
-0#define RV_FENTRY_NINSNS 2@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,emit_or (RV_REG_T1 , rd,RV_REG_T1 , ctx);emit_mv (rd,RV_REG_T1 , ctx);break ; + }else if (insn_is_mov_percpu_addr (insn)) { +if (rd != rs) +emit_mv (rd, rs, ctx);Is this an unconditional move instruction? in x86-64,
EMIT_movchecks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?
Quoted 2 lines
Yeah, it is. Folding the check into the emit sounds like a good idea.
great
Quoted 2 lines
Björn