Skip to contentSkip to conversation

riscv, bpf: Support per-CPU insn and inline bpf_get_smp_processor_id()

PATCH In review Puranjay Mohan 5 participants 11 messages Apr 30, 2024 – May 2, 2024

Subsystems (4)
lore
Share & tools
lore

Puranjay Mohan

author reviewer
#0 lore raw

v2 cover riscv, bpf: Support per-CPU insn and inline bpf_get_smp_processor_id()

Changes in v1->v2:
v1: https://lore.kernel.org/all/[email protected]/
- Use emit_addr() in place of emit_imm() in the first patch.
- Add the second patch to inline bpf_get_smp_processor_id()

The first patch add the support for the internal-only MOV instruction to resolve per-CPU addrs to absolute addresses.
RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

The second patch inlines the calls bpf_get_smp_processor_id() in the JIT by emitting a load from the TP + offsetof(struct thread_info, cpu).

Benchmark using [1] on Qemu.

./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc

  +---------------+------------------+------------------+--------------+
  |      Name     |     Before       |       After      |   % change   |
  |---------------+------------------+------------------+--------------|
  | glob-arr-inc  | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s |   + 24.04%   |
  | arr-inc       | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s |   + 23.56%   |
  | hash-inc      | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s |   + 32.18%   |
  +---------------+------------------+------------------+--------------+

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

Puranjay Mohan (2):
riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs riscv, bpf: inline bpf_get_smp_processor_id()

 arch/riscv/net/bpf_jit_comp64.c | 50 +++++++++++++++++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               | 11 ++++++++
 kernel/bpf/verifier.c           |  2 ++
 4 files changed, 64 insertions(+)
Signature · 1 line
2.40.1
author reviewer
#1 lore raw

PATCH v2 1/2 riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +24 -0
index 15e482f2c657..99d7006f1420 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 			emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
 			emit_mv(rd, RV_REG_T1, ctx);
 			break;
+		} else if (insn_is_mov_percpu_addr(insn)) {
+			if (rd != rs)
+				emit_mv(rd, rs, ctx);
+#ifdef CONFIG_SMP
+				/* Load current CPU number in T1 */
+				emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu),
+					RV_REG_TP, ctx);
+				/* << 3 because offsets are 8 bytes */
+				emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+				/* Load address of __per_cpu_offset array in T2 */
+				emit_addr(RV_REG_T2, (u64)&__per_cpu_offset, extra_pass, ctx);
+				/* Add offset of current CPU to  __per_cpu_offset */
+				emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+				/* Load __per_cpu_offset[cpu] in T1 */
+				emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+				/* Add the offset to Rd */
+				emit_add(rd, rd, RV_REG_T1, ctx);
+#endif
 		}
 		if (imm == 1) {
 			/* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void)
 {
 	return true;
 }
+
+bool bpf_jit_supports_percpu_insn(void)
+{
+	return true;
+}
Signature · 1 line
2.40.1
author reviewer
#5 lore raw

PATCH v2 2/2 riscv, bpf: inline bpf_get_smp_processor_id()

In reply to Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

          RISCV64 JIT output for `call bpf_get_smp_processor_id`
	  ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Benchmark using [1] on Qemu.

./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc

+---------------+------------------+------------------+--------------+
|      Name     |     Before       |       After      |   % change   |
|---------------+------------------+------------------+--------------|
| glob-arr-inc  | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s |   + 24.04%   |
| arr-inc       | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s |   + 23.56%   |
| hash-inc      | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s |   + 32.18%   |
+---------------+------------------+------------------+--------------+

NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 26 ++++++++++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               | 11 +++++++++++
 kernel/bpf/verifier.c           |  2 ++
 4 files changed, 40 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +26 -0
index 99d7006f1420..5789b7afae47 100644
@@ -1493,6 +1493,22 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
 		bool fixed_addr;
 		u64 addr;
 
+		/* Inline calls to bpf_get_smp_processor_id()
+		 *
+		 * RV_REG_TP holds the address of the current CPU's task_struct and thread_info is
+		 * at offset 0 in task_struct.
+		 * Load cpu from thread_info:
+		 *     Set R0 to ((struct thread_info *)(RV_REG_TP))->cpu
+		 *
+		 * This replicates the implementation of raw_smp_processor_id() on RISCV
+		 */
+		if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
+			/* Load current CPU number in R0 */
+			emit_ld(bpf_to_rv_reg(BPF_REG_0, ctx), offsetof(struct thread_info, cpu),
+				RV_REG_TP, ctx);
+			break;
+		}
+
 		mark_call(ctx);
 		ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
 					    &addr, &fixed_addr);
@@ -2062,3 +2078,13 @@ bool bpf_jit_supports_percpu_insn(void)
 {
 	return true;
 }
+
+bool bpf_jit_inlines_helper_call(s32 imm)
+{
+	switch (imm) {
+	case BPF_FUNC_get_smp_processor_id:
+		return true;
+	}
+
+	return false;
+}
include/linux/filter.h +1 -0
index 7a27f19bf44d..3e19bb62ed1a 100644
@@ -993,6 +993,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);
 void bpf_jit_compile(struct bpf_prog *prog);
 bool bpf_jit_needs_zext(void);
+bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
kernel/bpf/core.c +11 -0
index 99b8b1c9a248..aa59af9f9bd9 100644
@@ -2941,6 +2941,17 @@ bool __weak bpf_jit_needs_zext(void)
 	return false;
 }
 
+/* Return true if the JIT inlines the call to the helper corresponding to
+ * the imm.
+ *
+ * The verifier will not patch the insn->imm for the call to the helper if
+ * this returns true.
+ */
+bool __weak bpf_jit_inlines_helper_call(s32 imm)
+{
+	return false;
+}
+
 /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool __weak bpf_jit_supports_subprog_tailcalls(void)
 {
kernel/bpf/verifier.c +2 -0
index 5d42db05315e..e78f766d7f91 100644
@@ -20013,6 +20013,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
 			goto next_insn;
 		}
 
+		if (bpf_jit_inlines_helper_call(insn->imm))
+			goto next_insn;
 		if (insn->imm == BPF_FUNC_get_route_realm)
 			prog->dst_needed = 1;
 		if (insn->imm == BPF_FUNC_get_prandom_u32)
Signature · 1 line
2.40.1
maintainer Acked-by on v2 2/2
#6 lore raw

Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()

Quoted 30 lines · Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

          RISCV64 JIT output for `call bpf_get_smp_processor_id`
          ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Benchmark using [1] on Qemu.

./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc

+---------------+------------------+------------------+--------------+
|      Name     |     Before       |       After      |   % change   |
|---------------+------------------+------------------+--------------|
| glob-arr-inc  | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s |   + 24.04%   |
| arr-inc       | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s |   + 23.56%   |
| hash-inc      | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s |   + 32.18%   |
+---------------+------------------+------------------+--------------+
Quoted 7 lines

NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---

For non-riscv bits (& fwiw):

#2 lore raw

Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

In reply to Puranjay Mohan
Quoted 60 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +8 -0
index 15e482f2c657..99d7006f1420 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);
+#ifdef CONFIG_SMP
+                               /* Load current CPU number in T1 */
+                               emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu),
+                                       RV_REG_TP, ctx);

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1109

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+                               /* << 3 because offsets are 8 bytes */
+                               emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+                               /* Load address of __per_cpu_offset array in T2 */
+                               emit_addr(RV_REG_T2, (u64)&__per_cpu_offset, extra_pass, ctx);
+                               /* Add offset of current CPU to  __per_cpu_offset */
+                               emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+                               /* Load __per_cpu_offset[cpu] in T1 */
+                               emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+                               /* Add the offset to Rd */
+                               emit_add(rd, rd, RV_REG_T1, ctx);

is this the right level of code indentation?

Quoted 16 lines
arch/riscv/net/bpf_jit_comp64.c +6 -1
+#endif
                }
                if (imm == 1) {
                        /* Special mov32 for zext */
@@ -2038,3 +2057,8 @@ bool bpf_jit_supports_arena(void)
 {
        return true;
 }
+
+bool bpf_jit_supports_percpu_insn(void)
+{
+       return true;
+}
--

2.40.1

Acked-by on v2 2/2
#7 lore raw

Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()

In reply to Puranjay Mohan
Quoted 8 lines · Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

Quoted 10 lines
          RISCV64 JIT output for `call bpf_get_smp_processor_id`
          ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling bpf_get_smp_processor_id for x86-64 shows this:

Dump of assembler code for function bpf_get_smp_processor_id:
0xffffffff810f91a0 <+0>: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0xffffffff810f91a5 <+5>: 65 8b 05 60 79 f3 7e mov
%gs:0x7ef37960(%rip),%eax # 0x30b0c <pcpu_hot+12>

   0xffffffff810f91ac <+12>:    48 98   cltq
   0xffffffff810f91ae <+14>:    c3      ret

End of assembler dump.

We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").

Anyways, great work, a small nit below.

Quoted 55 lines

Benchmark using [1] on Qemu.

./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc

+---------------+------------------+------------------+--------------+
|      Name     |     Before       |       After      |   % change   |
|---------------+------------------+------------------+--------------|
| glob-arr-inc  | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s |   + 24.04%   |
| arr-inc       | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s |   + 23.56%   |
| hash-inc      | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s |   + 32.18%   |
+---------------+------------------+------------------+--------------+

NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 26 ++++++++++++++++++++++++++
 include/linux/filter.h          |  1 +
 kernel/bpf/core.c               | 11 +++++++++++
 kernel/bpf/verifier.c           |  2 ++
 4 files changed, 40 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +16 -0
index 99d7006f1420..5789b7afae47 100644
@@ -1493,6 +1493,22 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                bool fixed_addr;
                u64 addr;
 
+               /* Inline calls to bpf_get_smp_processor_id()
+                *
+                * RV_REG_TP holds the address of the current CPU's task_struct and thread_info is
+                * at offset 0 in task_struct.
+                * Load cpu from thread_info:
+                *     Set R0 to ((struct thread_info *)(RV_REG_TP))->cpu
+                *
+                * This replicates the implementation of raw_smp_processor_id() on RISCV
+                */
+               if (insn->src_reg == 0 && insn->imm == BPF_FUNC_get_smp_processor_id) {
+                       /* Load current CPU number in R0 */
+                       emit_ld(bpf_to_rv_reg(BPF_REG_0, ctx), offsetof(struct thread_info, cpu),
+                               RV_REG_TP, ctx);
+                       break;
+               }
+
                mark_call(ctx);
                ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
                                            &addr, &fixed_addr);
@@ -2062,3 +2078,13 @@ bool bpf_jit_supports_percpu_insn(void)
 {
        return true;

arch/riscv/net/bpf_jit_comp64.c bool bpf_jit_supports_percpu_insn(void) · line 2089

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +9 -0
 }
+
+bool bpf_jit_inlines_helper_call(s32 imm)
+{
+       switch (imm) {
+       case BPF_FUNC_get_smp_processor_id:
+               return true;
+       }
+
+       return false;

nit: why not

default:

    return false;

to keep everything within the switch?

Quoted 40 lines
arch/riscv/net/bpf_jit_comp64.c +1 -0
+}
include/linux/filter.h +1 -0
index 7a27f19bf44d..3e19bb62ed1a 100644
@@ -993,6 +993,7 @@ u64 __bpf_call_base(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog);
 void bpf_jit_compile(struct bpf_prog *prog);
 bool bpf_jit_needs_zext(void);
+bool bpf_jit_inlines_helper_call(s32 imm);
 bool bpf_jit_supports_subprog_tailcalls(void);
 bool bpf_jit_supports_percpu_insn(void);
 bool bpf_jit_supports_kfunc_call(void);
kernel/bpf/core.c +11 -0
index 99b8b1c9a248..aa59af9f9bd9 100644
@@ -2941,6 +2941,17 @@ bool __weak bpf_jit_needs_zext(void)
        return false;
 }
 
+/* Return true if the JIT inlines the call to the helper corresponding to
+ * the imm.
+ *
+ * The verifier will not patch the insn->imm for the call to the helper if
+ * this returns true.
+ */
+bool __weak bpf_jit_inlines_helper_call(s32 imm)
+{
+       return false;
+}
+
 /* Return TRUE if the JIT backend supports mixing bpf2bpf and tailcalls. */
 bool __weak bpf_jit_supports_subprog_tailcalls(void)
 {
kernel/bpf/verifier.c +0 -0
index 5d42db05315e..e78f766d7f91 100644

kernel/bpf/verifier.c static int do_misc_fixups(struct bpf_verifier_env *env) · line 20017

Quoted 5 lines
kernel/bpf/verifier.c +2 -0
@@ -20013,6 +20013,8 @@ static int do_misc_fixups(struct bpf_verifier_env *env)
                        goto next_insn;
                }
 
+               if (bpf_jit_inlines_helper_call(insn->imm))
+                       goto next_insn;

It's nice to be able to allow BPF JIT to do a higher-performance implementation. Let's add a short comment above to mention that this is bypassing normal inlining because BPF JIT will do it better (I know you have this description for the function definition, but a short remark here would be helpful).

And please add an empty line after this check to logically separate it from the rest of helper inlining logic in verifier, thanks!

pw-bot: cr

Quoted 6 lines
kernel/bpf/verifier.c +0 -1
                if (insn->imm == BPF_FUNC_get_route_realm)
                        prog->dst_needed = 1;
                if (insn->imm == BPF_FUNC_get_prandom_u32)
--

2.40.1

author reviewer
#8 lore raw

Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()

Quoted 29 lines · Andrii Nakryiko
Quoted 8 lines · Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

Quoted 10 lines
          RISCV64 JIT output for `call bpf_get_smp_processor_id`
          ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling bpf_get_smp_processor_id for x86-64 shows this:

Dump of assembler code for function bpf_get_smp_processor_id:

   0xffffffff810f91a0 <+0>:     0f 1f 44 00 00  nopl   0x0(%rax,%rax,1)
   0xffffffff810f91a5 <+5>:     65 8b 05 60 79 f3 7e    mov
%gs:0x7ef37960(%rip),%eax        # 0x30b0c <pcpu_hot+12>
   0xffffffff810f91ac <+12>:    48 98   cltq
   0xffffffff810f91ae <+14>:    c3      ret
Quoted 4 lines

End of assembler dump.
We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").

I feel in x86-64's case JIT can not do a (much) better job compared to the current approach in the verifier.

On RISC-V and ARM64, JIT was able to do it better because both of these architectures save a pointer to the task struct in a special CPU register. As x86-64 doesn't have enough extra registers, it uses a percpu variable to store task struct, thread_info, and the cpu number.

P.S. - While doing this for BPF, I realized that ARM64 kernel code is also not optimal as it is using the percpu variable and is not reading the CPU register directly. So, I sent a patch[1] to fix it in the kernel and get rid of the per-cpu variable in ARM64.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

Anyways, great work, a small nit below.

Thanks,
Puranjay

#9 lore raw

Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()

Quoted 37 lines · Puranjay Mohan
Quoted 29 lines · Andrii Nakryiko
Quoted 8 lines · Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

Quoted 10 lines
          RISCV64 JIT output for `call bpf_get_smp_processor_id`
          ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Nice, great find! Would you be able to do similar inlining for x86-64 as well? Disassembling bpf_get_smp_processor_id for x86-64 shows this:

Dump of assembler code for function bpf_get_smp_processor_id:

   0xffffffff810f91a0 <+0>:     0f 1f 44 00 00  nopl   0x0(%rax,%rax,1)
   0xffffffff810f91a5 <+5>:     65 8b 05 60 79 f3 7e    mov
%gs:0x7ef37960(%rip),%eax        # 0x30b0c <pcpu_hot+12>
   0xffffffff810f91ac <+12>:    48 98   cltq
   0xffffffff810f91ae <+14>:    c3      ret
Quoted 4 lines

End of assembler dump.
We should be able to do the same in x86-64 BPF JIT. (it's actually how I started initially, I had a dedicated instruction reading per-cpu memory, but ended up with more general "calculate per-cpu address").

Quoted 2 lines

I feel in x86-64's case JIT can not do a (much) better job compared to the current approach in the verifier.

This direct memory read (using gs segment) ought to be a bit faster than calculating offset and then doing memory dereference, but yes, the difference won't be as big as you got with RISC-V and ARM64. Ok, never mind, we can always benchmark and add that later, no big deal.

Quoted 21 lines

On RISC-V and ARM64, JIT was able to do it better because both of these architectures save a pointer to the task struct in a special CPU register. As x86-64 doesn't have enough extra registers, it uses a percpu variable to store task struct, thread_info, and the cpu number.

P.S. - While doing this for BPF, I realized that ARM64 kernel code is also not optimal as it is using the percpu variable and is not reading the CPU register directly. So, I sent a patch[1] to fix it in the kernel and get rid of the per-cpu variable in ARM64.

[1] https://lore.kernel.org/all/[email protected]/

Quoted 3 lines

Anyways, great work, a small nit below.

Thanks,
Puranjay

maintainer Acked-by
#3 lore raw

Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

In reply to Andrii Nakryiko
Quoted 49 lines · Andrii Nakryiko
Quoted 48 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +1 -0
index 15e482f2c657..99d7006f1420 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
Quoted 8 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095

Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);

No biggie, but you did not fold this check into emit_mv().

Quoted 15 lines
Quoted 4 lines
arch/riscv/net/bpf_jit_comp64.c +4 -0
+#ifdef CONFIG_SMP
+                               /* Load current CPU number in T1 */
+                               emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu),
+                                       RV_REG_TP, ctx);

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1109

Quoted 10 lines
arch/riscv/net/bpf_jit_comp64.c +10 -0
+                               /* << 3 because offsets are 8 bytes */
+                               emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+                               /* Load address of __per_cpu_offset array in T2 */
+                               emit_addr(RV_REG_T2, (u64)&__per_cpu_offset, extra_pass, ctx);
+                               /* Add offset of current CPU to  __per_cpu_offset */
+                               emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+                               /* Load __per_cpu_offset[cpu] in T1 */
+                               emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+                               /* Add the offset to Rd */
+                               emit_add(rd, rd, RV_REG_T1, ctx);
Quoted 1 line

is this the right level of code indentation?

Looks wrong.

When the indent is fixed, feel free to add:

maintainer Acked-by on v2 2/2
#10 lore raw

Re: [PATCH bpf-next v2 2/2] riscv, bpf: inline bpf_get_smp_processor_id()

In reply to Puranjay Mohan
Quoted 29 lines · Puranjay Mohan

Inline the calls to bpf_get_smp_processor_id() in the riscv bpf jit.

RISCV saves the pointer to the CPU's task_struct in the TP (thread pointer) register. This makes it trivial to get the CPU's processor id. As thread_info is the first member of task_struct, we can read the processor id from TP + offsetof(struct thread_info, cpu).

          RISCV64 JIT output for `call bpf_get_smp_processor_id`
	  ======================================================

                Before                           After
               --------                         -------

         auipc   t1,0x848c                  ld    a5,32(tp)
         jalr    604(t1)
         mv      a5,a0

Benchmark using [1] on Qemu.

./benchs/run_bench_trigger.sh glob-arr-inc arr-inc hash-inc

+---------------+------------------+------------------+--------------+
|      Name     |     Before       |       After      |   % change   |
|---------------+------------------+------------------+--------------|
| glob-arr-inc  | 1.077 ± 0.006M/s | 1.336 ± 0.010M/s |   + 24.04%   |
| arr-inc       | 1.078 ± 0.002M/s | 1.332 ± 0.015M/s |   + 23.56%   |
| hash-inc      | 0.494 ± 0.004M/s | 0.653 ± 0.001M/s |   + 32.18%   |
+---------------+------------------+------------------+--------------+
Quoted 6 lines

NOTE: This benchmark includes changes from this patch and the previous
patch that implemented the per-cpu insn.

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

#4 lore raw

Re: [PATCH bpf-next v2 1/2] riscv, bpf: add internal-only MOV instruction to resolve per-CPU addrs

In reply to Björn Töpel
Quoted 78 lines · Björn Töpel
Quoted 49 lines · Andrii Nakryiko
Quoted 48 lines · Puranjay Mohan

Support an instruction for resolving absolute addresses of per-CPU data from their per-CPU offsets. This instruction is internal-only and users are not allowed to use them directly. They will only be used for internal inlining optimizations for now between BPF verifier and BPF JITs.

RISC-V uses generic per-cpu implementation where the offsets for CPUs are kept in an array called __per_cpu_offset[cpu_number]. RISCV stores the address of the task_struct in TP register. The first element in task_struct is struct thread_info, and we can get the cpu number by reading from the TP register + offsetof(struct thread_info, cpu).

Once we have the cpu number in a register we read the offset for that cpu from address: &__per_cpu_offset + cpu_number << 3. Then we add this offset to the destination register.

To measure the improvement from this change, the benchmark in [1] was used on Qemu:

Before:

glob-arr-inc   :    1.127 ± 0.013M/s
arr-inc        :    1.121 ± 0.004M/s
hash-inc       :    0.681 ± 0.052M/s

After:

glob-arr-inc   :    1.138 ± 0.011M/s
arr-inc        :    1.366 ± 0.006M/s
hash-inc       :    0.676 ± 0.001M/s

[1] https://github.com/anakryiko/linux/commit/8dec900975ef

---
 arch/riscv/net/bpf_jit_comp64.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
arch/riscv/net/bpf_jit_comp64.c +1 -0
index 15e482f2c657..99d7006f1420 100644
@@ -12,6 +12,7 @@
 #include <linux/stop_machine.h>
 #include <asm/patch.h>
 #include <asm/cfi.h>
+#include <asm/percpu.h>
 #include "bpf_jit.h"
 
Quoted 8 lines

arch/riscv/net/bpf_jit_comp64.c int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · line 1095

Quoted 8 lines
arch/riscv/net/bpf_jit_comp64.c +3 -0
 #define RV_FENTRY_NINSNS 2
@@ -1089,6 +1090,24 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
                        emit_or(RV_REG_T1, rd, RV_REG_T1, ctx);
                        emit_mv(rd, RV_REG_T1, ctx);
                        break;
+               } else if (insn_is_mov_percpu_addr(insn)) {
+                       if (rd != rs)
+                               emit_mv(rd, rs, ctx);

No biggie, but you did not fold this check into emit_mv().

Quoted 15 lines
Quoted 14 lines
arch/riscv/net/bpf_jit_comp64.c +14 -0
+#ifdef CONFIG_SMP
+                               /* Load current CPU number in T1 */
+                               emit_ld(RV_REG_T1, offsetof(struct thread_info, cpu),
+                                       RV_REG_TP, ctx);
+                               /* << 3 because offsets are 8 bytes */
+                               emit_slli(RV_REG_T1, RV_REG_T1, 3, ctx);
+                               /* Load address of __per_cpu_offset array in T2 */
+                               emit_addr(RV_REG_T2, (u64)&__per_cpu_offset, extra_pass, ctx);
+                               /* Add offset of current CPU to  __per_cpu_offset */
+                               emit_add(RV_REG_T1, RV_REG_T2, RV_REG_T1, ctx);
+                               /* Load __per_cpu_offset[cpu] in T1 */
+                               emit_ld(RV_REG_T1, 0, RV_REG_T1, ctx);
+                               /* Add the offset to Rd */
+                               emit_add(rd, rd, RV_REG_T1, ctx);
Quoted 7 lines
Quoted 1 line

is this the right level of code indentation?

Looks wrong.

When the indent is fixed, feel free to add:

I fixed the indent and sent it as part of:
https://lore.kernel.org/all/[email protected]/

Also, for the emit_mv() thing, I wanted to verify if we are using that somewhere for zero-extension or something.
So, I thought I would send a separate patch for it.

Thanks,
Puranjay