Skip to contentSkip to conversation

riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

PATCH In review Puranjay Mohan 3 participants 6 messages Apr 5, 2024 – Apr 18, 2024

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +24 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
 			emit_mv(rd, RV_REG_T1, ctx);
 			break;
+		} else if (insn_is_mov_percpu_addr(insn)) {
+			if (rd != rs)
+				emit_mv(rd, rs, ctx);
+#ifdef CONFIG_SMP
+				/* Load current CPU number in T1 */
+				emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP,
+					ctx);
+				/* << 3 because offsets are 8 bytes */
+				emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+				/* Load address of __per_cpu_offset array in T2 */
+				emit_imm(RV_REG_T2, (u64)&__per_cpu_offset, ctx);
+				/* Add offset of current CPU to  __per_cpu_offset */
+				emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+				/* Load __per_cpu_offset[cpu] in T1 */
+				emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+				/* Add the offset to Rd */
+				emit_add(rd, rd, RV_REG_T1, ctx);
+#endif
 		}
 		if (imm == 1) {
 			/* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_percpu_insn(void)
+{
+	return true;
+}
Signature · 1 line
2.40.1
maintainer
#1 lore raw
Quoted 10 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by

     ^
     k ;-)
Quoted 5 lines

reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

Just to clarify for readers; BPF programs are run with migrate disable, which means that on RT we can be preempted, which means that per-cpu operations are trickier (disabling interrupts/preemption).

However, this BPF instruction is about calculating the per-cpu address, so the look up can be inlined.

It's not a per-cpu *operation*.

Quoted 37 lines

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +2 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
 			emit_mv(rd, RV_REG_T1, ctx);
 			break;
+		} else if (insn_is_mov_percpu_addr(insn)) {

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1103

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+			if (rd != rs)
+				emit_mv(rd, rs, ctx);
+#ifdef CONFIG_SMP
+				/* Load current CPU number in T1 */
+				emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP,
+					ctx);
+				/* << 3 because offsets are 8 bytes */
+				emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+				/* Load address of __per_cpu_offset array in T2 */
+				emit_imm(RV_REG_T2, (u64)&__per_cpu_offset, ctx);

Did you try using emit_addr() here? I'd guess that'll be fewer instructions, no?

Björn

#2 lore raw
Quoted 8 lines · Björn Töpel
Quoted 6 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

Quoted 6 lines
Quoted 4 lines

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by

     ^
     k ;-)

I need to start using some kind of spell-check in vim :D.

Quoted 6 lines
Quoted 5 lines

reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

Quoted 8 lines

Just to clarify for readers; BPF programs are run with migrate disable, which means that on RT we can be preempted, which means that per-cpu operations are trickier (disabling interrupts/preemption).

However, this BPF instruction is about calculating the per-cpu address, so the look up can be inlined.

It's not a per-cpu *operation*.

Will add this information to the commit message.

Quoted 42 lines
Quoted 42 lines

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +7 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
 			emit_mv(rd, RV_REG_T1, ctx);
 			break;
+		} else if (insn_is_mov_percpu_addr(insn)) {
+			if (rd != rs)
+				emit_mv(rd, rs, ctx);
+#ifdef CONFIG_SMP
+				/* Load current CPU number in T1 */
+				emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP,
Quoted 8 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1103

Quoted 5 lines
arch/riscv/net/bpf_jit_comp64.c +5 -0
+					ctx);
+				/* << 3 because offsets are 8 bytes */
+				emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+				/* Load address of __per_cpu_offset array in T2 */
+				emit_imm(RV_REG_T2, (u64)&__per_cpu_offset, ctx);

Did you try using emit_addr() here? I'd guess that'll be fewer instructions, no?

Yes, I should have used that, the address would always be in the range of auipc+addi right? I will try and see.

Thanks,
Puranjay

#3 lore raw
In reply to Puranjay Mohan
Quoted 21 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

Quoted 8 lines
glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

TBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?

Quoted 21 lines

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +1 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095

Quoted 6 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);

Is this an unconditional move instruction? in x86-64, EMIT_mov checks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1099

Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+#ifdef CONFIG_SMP
+                               /* Load current CPU number in T1 */
+                               emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu), RV_REG_TP,
+                                       ctx);

nit: maybe keep this on the same line?

Quoted 26 lines
arch/riscv/net/bpf_jit_comp64.c +16 -1
+                               /* << 3 because offsets are 8 bytes */
+                               emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+                               /* Load address of __per_cpu_offset array in T2 */
+                               emit_imm(RV_REG_T2, (u64)&__per_cpu_offset, ctx);
+                               /* Add offset of current CPU to  __per_cpu_offset */
+                               emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+                               /* Load __per_cpu_offset[cpu] in T1 */
+                               emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+                               /* Add the offset to Rd */
+                               emit_add(rd, rd, RV_REG_T1, ctx);
+#endif
                }
                if (imm == 1) {
                        /* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void)
 {
        return true;
 }
+
+bool bpf_jit_supports_percpu_insn(void)
+{
+       return true;
+}
--

2.40.1

maintainer
#4 lore raw
Quoted 26 lines · Andrii Nakryiko
Quoted 25 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s
Quoted 7 lines
Quoted 4 lines

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

TBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?

I just ran it on a "VisionFive2" SBC:

BEFORE
======

glob-arr-inc   :   11.586 ± 0.021M/s 
arr-inc        :   10.892 ± 0.005M/s 
hash-inc       :    1.517 ± 0.001M/s 

AFTER
=====

glob-arr-inc   :   11.893 ± 0.017M/s  (+2.6%)
arr-inc        :   11.630 ± 0.020M/s  (+6.8%)
hash-inc       :    1.543 ± 0.002M/s  (+1.7%)

(It's early, and the coffee haven't kicked in, so I hope the calculations are correct...)

Quoted 19 lines
Quoted 19 lines

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +1 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
Quoted 12 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095

Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);

Is this an unconditional move instruction? in x86-64, EMIT_mov checks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?

Yeah, it is. Folding the check into the emit sounds like a good idea.

Björn

#5 lore raw
Quoted 45 lines · Björn Töpel
Quoted 26 lines · Andrii Nakryiko
Quoted 25 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in tast_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s
Quoted 7 lines
Quoted 4 lines

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

TBH, I don't trust benchmarks done inside QEMU. Can you try running this on some real hardware?

I just ran it on a "VisionFive2" SBC:

BEFORE
======

glob-arr-inc   :   11.586 ± 0.021M/s
arr-inc        :   10.892 ± 0.005M/s
hash-inc       :    1.517 ± 0.001M/s
Quoted 6 lines

AFTER
=====

glob-arr-inc   :   11.893 ± 0.017M/s  (+2.6%)
arr-inc        :   11.630 ± 0.020M/s  (+6.8%)
hash-inc       :    1.543 ± 0.002M/s  (+1.7%)

Nice, looks pretty reasonable (and especially if bpf_smp_get_current_id() gets inlined as well, the numbers should be even better)

Quoted 35 lines

(It's early, and the coffee haven't kicked in, so I hope the calculations are correct...)

Quoted 19 lines
Quoted 19 lines

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +1 -0
index 15e482f2c657..e95bd1d459a4 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
Quoted 12 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095

Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);

Is this an unconditional move instruction? in x86-64, EMIT_mov checks whether source and destination registers are the same and doesn't emit anything if they match (which makes sense, right)?

Quoted 2 lines

Yeah, it is. Folding the check into the emit sounds like a good idea.

great

Quoted 2 lines

Björn