Skip to contentSkip to conversation

powerpc/bpf: use BPF prog pack allocator

PATCH New Hari Bathini 2 participants 11 messages Aug 25, 2023

Subsystems (3)
lore
Share & tools
lore

Hari Bathini

author maintainer
#0 lore raw

cover powerpc/bpf: use BPF prog pack allocator

Most BPF programs are small, but they consume a page each. For systems with busy traffic and many BPF programs, this may also add significant pressure on instruction TLB. High iTLB pressure usually slows down the whole system causing visible performance degradation for production workloads.

bpf_prog_pack, a customized allocator that packs multiple bpf programs into preallocated memory chunks, was proposed [1] to address it. This series extends this support on powerpc.

Patches 1 & 2 add the arch specific functions needed to support this feature. Patch 3 enables the support for powerpc and ensures cleanup is handled gracefully. Patch 4 introduces patch_instructions() that optimizes some calls while patching more than one instruction. Patch 5 leverages this new function to improve time taken for JIT'ing BPF programs.

Note that the first 3 patches are sufficient to enable the support for bpf_prog_pack on powerpc. Patches 4 & 5 are to improve the JIT compilation time of BPF programs on powerpc.

Thanks to Christophe and Song for reviewing v2 [2].

Changes in v3:
* Fixed segfault issue observed on ppc32 due to inaccurate offset calculation for branching.
* Tried to minimize the performance impact for patch_instruction() with the introduction of patch_instructions().
* Corrected uses of u32* vs ppc_instr_t.
* Moved the change that introduces patch_instructions() to after enabling bpf_prog_pack support.
* Added few comments to improve code readability.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Hari Bathini (5):
powerpc/bpf: implement bpf_arch_text_copy
powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free] powerpc/code-patching: introduce patch_instructions() powerpc/bpf: use patch_instructions()

 arch/powerpc/include/asm/code-patching.h |   1 +
 arch/powerpc/lib/code-patching.c         |  94 ++++++++++++---
 arch/powerpc/net/bpf_jit.h               |  12 +-
 arch/powerpc/net/bpf_jit_comp.c          | 138 ++++++++++++++++++-----
 arch/powerpc/net/bpf_jit_comp32.c        |  13 ++-
 arch/powerpc/net/bpf_jit_comp64.c        |  10 +-
 6 files changed, 205 insertions(+), 63 deletions(-)
Signature · 1 line
2.41.0
author maintainer
#1 lore raw

PATCH v3 1/5 powerpc/bpf: implement bpf_arch_text_copy

bpf_arch_text_copy is used to dump JITed binary to RX page, allowing multiple BPF programs to share the same page. Use patch_instruction() to implement it.

---
 arch/powerpc/net/bpf_jit_comp.c | 40 ++++++++++++++++++++++++++++++++-
 1 file changed, 39 insertions(+), 1 deletion(-)
arch/powerpc/net/bpf_jit_comp.c +39 -1
index 37043dfc1add..170ebf8ac0f2 100644
@@ -13,9 +13,12 @@
 #include <linux/netdevice.h>
 #include <linux/filter.h>
 #include <linux/if_vlan.h>
-#include <asm/kprobes.h>
+#include <linux/memory.h>
 #include <linux/bpf.h>
 
+#include <asm/kprobes.h>
+#include <asm/code-patching.h>
+
 #include "bpf_jit.h"
 
 static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
@@ -23,6 +26,27 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
 	memset32(area, BREAKPOINT_INSTRUCTION, size / 4);
 }
 
+/*
+ * Patch 'len' bytes of instructions from opcode to addr, one instruction
+ * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
+ */
+static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
+{
+	while (len > 0) {
+		ppc_inst_t insn = ppc_inst_read(opcode);
+		int ilen = ppc_inst_len(insn);
+
+		if (patch_instruction(addr, insn))
+			return ERR_PTR(-EINVAL);
+
+		len -= ilen;
+		addr = addr + ilen;
+		opcode = opcode + ilen;
+	}
+
+	return addr;
+}
+
 int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr)
 {
 	if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) {
@@ -274,3 +298,17 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct code
 	ctx->exentry_idx++;
 	return 0;
 }
+
+void *bpf_arch_text_copy(void *dst, void *src, size_t len)
+{
+	void *ret;
+
+	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
+		return ERR_PTR(-EINVAL);
+
+	mutex_lock(&text_mutex);
+	ret = bpf_patch_instructions(dst, src, len);
+	mutex_unlock(&text_mutex);
+
+	return ret;
+}
Signature · 1 line
2.41.0
#2 lore raw

Re: [PATCH v3 1/5] powerpc/bpf: implement bpf_arch_text_copy

Le 25/08/2023 à 17:18, Hari Bathini a écrit :

Quoted 3 lines

bpf_arch_text_copy is used to dump JITed binary to RX page, allowing multiple BPF programs to share the same page. Use patch_instruction() to implement it.

By using patch_instruction() for doing that you are mapping and unmapping the same page for each instruction you copy. You should implement something to map a page, copy all instruction going into that page, unmap the page and map next page, etc ....
And then perform the icache flush at the end.

Quoted 70 lines
---
  arch/powerpc/net/bpf_jit_comp.c | 40 ++++++++++++++++++++++++++++++++-
  1 file changed, 39 insertions(+), 1 deletion(-)
arch/powerpc/net/bpf_jit_comp.c +39 -1
index 37043dfc1add..170ebf8ac0f2 100644
@@ -13,9 +13,12 @@
  #include <linux/netdevice.h>
  #include <linux/filter.h>
  #include <linux/if_vlan.h>
-#include <asm/kprobes.h>
+#include <linux/memory.h>
  #include <linux/bpf.h>
 
+#include <asm/kprobes.h>
+#include <asm/code-patching.h>
+
  #include "bpf_jit.h"
 
  static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
@@ -23,6 +26,27 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
  	memset32(area, BREAKPOINT_INSTRUCTION, size / 4);
  }
 
+/*
+ * Patch 'len' bytes of instructions from opcode to addr, one instruction
+ * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
+ */
+static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
+{
+	while (len > 0) {
+		ppc_inst_t insn = ppc_inst_read(opcode);
+		int ilen = ppc_inst_len(insn);
+
+		if (patch_instruction(addr, insn))
+			return ERR_PTR(-EINVAL);
+
+		len -= ilen;
+		addr = addr + ilen;
+		opcode = opcode + ilen;
+	}
+
+	return addr;
+}
+
  int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr)
  {
  	if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) {
@@ -274,3 +298,17 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct code
  	ctx->exentry_idx++;
  	return 0;
  }
+
+void *bpf_arch_text_copy(void *dst, void *src, size_t len)
+{
+	void *ret;
+
+	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
+		return ERR_PTR(-EINVAL);
+
+	mutex_lock(&text_mutex);
+	ret = bpf_patch_instructions(dst, src, len);
+	mutex_unlock(&text_mutex);
+
+	return ret;
+}
author maintainer
#3 lore raw

PATCH v3 2/5 powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack

In reply to Hari Bathini

Implement bpf_arch_text_invalidate and use it to fill unused part of the bpf_prog_pack with trap instructions when a BPF program is freed.

---
 arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +19 -3
index 170ebf8ac0f2..7cd4cf53d61c 100644
@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
  * Patch 'len' bytes of instructions from opcode to addr, one instruction
  * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
  */
-static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
+static void *bpf_patch_instructions(void *addr, void *opcode, size_t len, bool fill_insn)
 {
 	while (len > 0) {
 		ppc_inst_t insn = ppc_inst_read(opcode);
@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
 
 		len -= ilen;
 		addr = addr + ilen;
-		opcode = opcode + ilen;
+		if (!fill_insn)
+			opcode = opcode + ilen;
 	}
 
 	return addr;
@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 		return ERR_PTR(-EINVAL);
 
 	mutex_lock(&text_mutex);
-	ret = bpf_patch_instructions(dst, src, len);
+	ret = bpf_patch_instructions(dst, src, len, false);
+	mutex_unlock(&text_mutex);
+
+	return ret;
+}
+
+int bpf_arch_text_invalidate(void *dst, size_t len)
+{
+	u32 insn = BREAKPOINT_INSTRUCTION;
+	int ret;
+
+	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
+		return -EINVAL;
+
+	mutex_lock(&text_mutex);
+	ret = IS_ERR(bpf_patch_instructions(dst, &insn, len, true));
 	mutex_unlock(&text_mutex);
 
 	return ret;
Signature · 1 line
2.41.0
#4 lore raw

Re: [PATCH v3 2/5] powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack

Le 25/08/2023 à 17:18, Hari Bathini a écrit :

Quoted 13 lines

Implement bpf_arch_text_invalidate and use it to fill unused part of the bpf_prog_pack with trap instructions when a BPF program is freed.

---
  arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++---
  1 file changed, 19 insertions(+), 3 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +0 -0
index 170ebf8ac0f2..7cd4cf53d61c 100644

arch/powerpc/net/bpf_jit_comp.c static void bpf_jit_fill_ill_insns(void *area, unsigned int size) · line 33

Quoted 5 lines
arch/powerpc/net/bpf_jit_comp.c +1 -1
@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
   * Patch 'len' bytes of instructions from opcode to addr, one instruction
   * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
   */
-static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
+static void *bpf_patch_instructions(void *addr, void *opcode, size_t len, bool fill_insn)

It's a pitty that you have to modify in patch 2 a function you have added in patch 1 of the same series. Can't you have it right from the begining ?

Quoted 24 lines
arch/powerpc/net/bpf_jit_comp.c +8 -2
  {
  	while (len > 0) {
  		ppc_inst_t insn = ppc_inst_read(opcode);
@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
 
  		len -= ilen;
  		addr = addr + ilen;
-		opcode = opcode + ilen;
+		if (!fill_insn)
+			opcode = opcode + ilen;
  	}
 
  	return addr;
@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)
  		return ERR_PTR(-EINVAL);
 
  	mutex_lock(&text_mutex);
-	ret = bpf_patch_instructions(dst, src, len);
+	ret = bpf_patch_instructions(dst, src, len, false);
+	mutex_unlock(&text_mutex);
+
+	return ret;
+}
+

arch/powerpc/net/bpf_jit_comp.c void *bpf_arch_text_copy(void *dst, void *src, size_t len) · line 326

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +10 -0
+int bpf_arch_text_invalidate(void *dst, size_t len)
+{
+	u32 insn = BREAKPOINT_INSTRUCTION;
+	int ret;
+
+	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
+		return -EINVAL;
+
+	mutex_lock(&text_mutex);
+	ret = IS_ERR(bpf_patch_instructions(dst, &insn, len, true));

Why IS_ERR ?

As far as I understand from the weak definition in kernel/bpf/core.c, this function is supposed to return an error, not a bool.

Quoted 3 lines
arch/powerpc/net/bpf_jit_comp.c +0 -0
  	mutex_unlock(&text_mutex);
 
  	return ret;
author maintainer
#5 lore raw

Re: [PATCH v3 2/5] powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack

Quoted 63 lines · Christophe Leroy

Le 25/08/2023 à 17:18, Hari Bathini a écrit :

Quoted 13 lines

Implement bpf_arch_text_invalidate and use it to fill unused part of the bpf_prog_pack with trap instructions when a BPF program is freed.

---
   arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++---
   1 file changed, 19 insertions(+), 3 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +0 -0
index 170ebf8ac0f2..7cd4cf53d61c 100644

arch/powerpc/net/bpf_jit_comp.c static void bpf_jit_fill_ill_insns(void *area, unsigned int size) · line 33

Quoted 5 lines
arch/powerpc/net/bpf_jit_comp.c +1 -1
@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
    * Patch 'len' bytes of instructions from opcode to addr, one instruction
    * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
    */
-static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
+static void *bpf_patch_instructions(void *addr, void *opcode, size_t len, bool fill_insn)

It's a pitty that you have to modify in patch 2 a function you have added in patch 1 of the same series. Can't you have it right from the begining ?

Quoted 24 lines
arch/powerpc/net/bpf_jit_comp.c +8 -2
   {
   	while (len > 0) {
   		ppc_inst_t insn = ppc_inst_read(opcode);
@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len)
 
   		len -= ilen;
   		addr = addr + ilen;
-		opcode = opcode + ilen;
+		if (!fill_insn)
+			opcode = opcode + ilen;
   	}
 
   	return addr;
@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)
   		return ERR_PTR(-EINVAL);
 
   	mutex_lock(&text_mutex);
-	ret = bpf_patch_instructions(dst, src, len);
+	ret = bpf_patch_instructions(dst, src, len, false);
+	mutex_unlock(&text_mutex);
+
+	return ret;
+}
+

arch/powerpc/net/bpf_jit_comp.c void *bpf_arch_text_copy(void *dst, void *src, size_t len) · line 326

Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +10 -0
+int bpf_arch_text_invalidate(void *dst, size_t len)
+{
+	u32 insn = BREAKPOINT_INSTRUCTION;
+	int ret;
+
+	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
+		return -EINVAL;
+
+	mutex_lock(&text_mutex);
+	ret = IS_ERR(bpf_patch_instructions(dst, &insn, len, true));

Why IS_ERR ?

Quoted 2 lines

As far as I understand from the weak definition in kernel/bpf/core.c, this function is supposed to return an error, not a bool.

My bad! Will fix that in the next revision.

- Hari

author maintainer
#6 lore raw

PATCH v3 3/5 powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]

In reply to Hari Bathini

Use bpf_jit_binary_pack_alloc in powerpc jit. The jit engine first writes the program to the rw buffer. When the jit is done, the program is copied to the final location with bpf_jit_binary_pack_finalize. With multiple jit_subprogs, bpf_jit_free is called on some subprograms that haven't got bpf_jit_binary_pack_finalize() yet. Implement custom bpf_jit_free() like in commit 1d5f82d9dd47 ("bpf, x86: fix freeing of not-finalized bpf_prog_pack") to call bpf_jit_binary_pack_finalize(), if necessary. While here, correct the misnomer powerpc64_jit_data to powerpc_jit_data as it is meant for both ppc32 and ppc64.

---
 arch/powerpc/net/bpf_jit.h        |  12 ++--
 arch/powerpc/net/bpf_jit_comp.c   | 110 ++++++++++++++++++++++--------
 arch/powerpc/net/bpf_jit_comp32.c |  13 ++--
 arch/powerpc/net/bpf_jit_comp64.c |  10 +--
 4 files changed, 98 insertions(+), 47 deletions(-)
arch/powerpc/net/bpf_jit.h +5 -7
index 72b7bb34fade..02bd89aa8ea5 100644
@@ -36,9 +36,6 @@
 		EMIT(PPC_RAW_BRANCH(offset));				      \
 	} while (0)
 
-/* bl (unconditional 'branch' with link) */
-#define PPC_BL(dest)	EMIT(PPC_RAW_BL((dest) - (unsigned long)(image + ctx->idx)))
-
 /* "cond" here covers BO:BI fields. */
 #define PPC_BCC_SHORT(cond, dest)					      \
 	do {								      \
@@ -169,16 +166,17 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i)
 }
 
 void bpf_jit_init_reg_mapping(struct codegen_context *ctx);
-int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func);
-int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
+int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func);
+int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct codegen_context *ctx,
 		       u32 *addrs, int pass, bool extra_pass);
 void bpf_jit_build_prologue(u32 *image, struct codegen_context *ctx);
 void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx);
 void bpf_jit_realloc_regs(struct codegen_context *ctx);
 int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr);
 
-int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct codegen_context *ctx,
-			  int insn_idx, int jmp_off, int dst_reg);
+int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass,
+			  struct codegen_context *ctx, int insn_idx,
+			  int jmp_off, int dst_reg);
 
 #endif
 
arch/powerpc/net/bpf_jit_comp.c +81 -29

220 lines not on this page · raw message

arch/powerpc/net/bpf_jit_comp32.c +7 -6
index 7f91ea064c08..434417c755fd 100644
@@ -200,12 +200,13 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
 	EMIT(PPC_RAW_BLR());
 }
 
-int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
+/* Relative offset needs to be calculated based on final image location */
+int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
 {
-	s32 rel = (s32)func - (s32)(image + ctx->idx);
+	s32 rel = (s32)func - (s32)(fimage + ctx->idx);
 
 	if (image && rel < 0x2000000 && rel >= -0x2000000) {
-		PPC_BL(func);
+		EMIT(PPC_RAW_BL(rel));
 	} else {
 		/* Load function address into r0 */
 		EMIT(PPC_RAW_LIS(_R0, IMM_H(func)));
@@ -278,7 +279,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o
 }
 
 /* Assemble the body code between the prologue & epilogue */
-int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
+int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct codegen_context *ctx,
 		       u32 *addrs, int pass, bool extra_pass)
 {
 	const struct bpf_insn *insn = fp->insnsi;
@@ -997,7 +998,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
 					jmp_off += 4;
 				}
 
-				ret = bpf_add_extable_entry(fp, image, pass, ctx, insn_idx,
+				ret = bpf_add_extable_entry(fp, image, fimage, pass, ctx, insn_idx,
 							    jmp_off, dst_reg);
 				if (ret)
 					return ret;
@@ -1053,7 +1054,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
 				EMIT(PPC_RAW_STW(bpf_to_ppc(BPF_REG_5), _R1, 12));
 			}
 
-			ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+			ret = bpf_jit_emit_func_call_rel(image, fimage, ctx, func_addr);
 			if (ret)
 				return ret;
 
arch/powerpc/net/bpf_jit_comp64.c +5 -5
index 0f8048f6dad6..79f23974a320 100644
@@ -240,7 +240,7 @@ static int bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx, u
 	return 0;
 }
 
-int bpf_jit_emit_func_call_rel(u32 *image, struct codegen_context *ctx, u64 func)
+int bpf_jit_emit_func_call_rel(u32 *image, u32 *fimage, struct codegen_context *ctx, u64 func)
 {
 	unsigned int i, ctx_idx = ctx->idx;
 
@@ -361,7 +361,7 @@ asm (
 );
 
 /* Assemble the body code between the prologue & epilogue */
-int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *ctx,
+int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, u32 *fimage, struct codegen_context *ctx,
 		       u32 *addrs, int pass, bool extra_pass)
 {
 	enum stf_barrier_type stf_barrier = stf_barrier_type_get();
@@ -940,8 +940,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
 				addrs[++i] = ctx->idx * 4;
 
 			if (BPF_MODE(code) == BPF_PROBE_MEM) {
-				ret = bpf_add_extable_entry(fp, image, pass, ctx, ctx->idx - 1,
-							    4, dst_reg);
+				ret = bpf_add_extable_entry(fp, image, fimage, pass, ctx,
+							    ctx->idx - 1, 4, dst_reg);
 				if (ret)
 					return ret;
 			}
@@ -995,7 +995,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *
 			if (func_addr_fixed)
 				ret = bpf_jit_emit_func_call_hlp(image, ctx, func_addr);
 			else
-				ret = bpf_jit_emit_func_call_rel(image, ctx, func_addr);
+				ret = bpf_jit_emit_func_call_rel(image, fimage, ctx, func_addr);
 
 			if (ret)
 				return ret;
Signature · 1 line
2.41.0
author maintainer
#7 lore raw

PATCH v3 4/5 powerpc/code-patching: introduce patch_instructions()

In reply to Hari Bathini

patch_instruction() entails setting up pte, patching the instruction, clearing the pte and flushing the tlb. If multiple instructions need to be patched, every instruction would have to go through the above drill unnecessarily. Instead, introduce function patch_instructions() that sets up the pte, clears the pte and flushes the tlb only once per page range of instructions to be patched. This adds a slight overhead to patch_instruction() call while improving the patching time for scenarios where more than one instruction needs to be patched.

---
 arch/powerpc/include/asm/code-patching.h |  1 +
 arch/powerpc/lib/code-patching.c         | 94 ++++++++++++++++++++----
 2 files changed, 80 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/code-patching.h +1 -0
index 3f881548fb61..4f5f0c091586 100644
@@ -74,6 +74,7 @@ int create_cond_branch(ppc_inst_t *instr, const u32 *addr,
 int patch_branch(u32 *addr, unsigned long target, int flags);
 int patch_instruction(u32 *addr, ppc_inst_t instr);
 int raw_patch_instruction(u32 *addr, ppc_inst_t instr);
+int patch_instructions(void *addr, void *code, size_t len, bool fill_insn);
 
 static inline unsigned long patch_site_addr(s32 *site)
 {
arch/powerpc/lib/code-patching.c +79 -15

148 lines not on this page · raw message

Signature · 1 line
2.41.0
author maintainer
#8 lore raw

PATCH v3 5/5 powerpc/bpf: use patch_instructions()

In reply to Hari Bathini

Use the newly introduced patch_instructions() that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.

Without this patch (on a POWER9 lpar):

  # time modprobe test_bpf
  real    2m59.681s
  user    0m0.000s
  sys     1m44.160s
  #

With this patch (on a POWER9 lpar):

  # time modprobe test_bpf
  real    0m5.013s
  user    0m0.000s
  sys     0m4.216s
  #
---
 arch/powerpc/net/bpf_jit_comp.c | 30 ++++--------------------------
 1 file changed, 4 insertions(+), 26 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +4 -26
index c60d7570e05d..1e5000d18321 100644
@@ -26,28 +26,6 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
 	memset32(area, BREAKPOINT_INSTRUCTION, size / 4);
 }
 
-/*
- * Patch 'len' bytes of instructions from opcode to addr, one instruction
- * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
- */
-static void *bpf_patch_instructions(void *addr, void *opcode, size_t len, bool fill_insn)
-{
-	while (len > 0) {
-		ppc_inst_t insn = ppc_inst_read(opcode);
-		int ilen = ppc_inst_len(insn);
-
-		if (patch_instruction(addr, insn))
-			return ERR_PTR(-EINVAL);
-
-		len -= ilen;
-		addr = addr + ilen;
-		if (!fill_insn)
-			opcode = opcode + ilen;
-	}
-
-	return addr;
-}
-
 int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr)
 {
 	if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) {
@@ -330,16 +308,16 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass
 
 void *bpf_arch_text_copy(void *dst, void *src, size_t len)
 {
-	void *ret;
+	int err;
 
 	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
 		return ERR_PTR(-EINVAL);
 
 	mutex_lock(&text_mutex);
-	ret = bpf_patch_instructions(dst, src, len, false);
+	err = patch_instructions(dst, src, len, false);
 	mutex_unlock(&text_mutex);
 
-	return ret;
+	return err ? ERR_PTR(err) : dst;
 }
 
 int bpf_arch_text_invalidate(void *dst, size_t len)
@@ -351,7 +329,7 @@ int bpf_arch_text_invalidate(void *dst, size_t len)
 		return -EINVAL;
 
 	mutex_lock(&text_mutex);
-	ret = IS_ERR(bpf_patch_instructions(dst, &insn, len, true));
+	ret = patch_instructions(dst, &insn, len, true);
 	mutex_unlock(&text_mutex);
 
 	return ret;
Signature · 1 line
2.41.0
#9 lore raw

Re: [PATCH v3 5/5] powerpc/bpf: use patch_instructions()

Le 25/08/2023 à 17:18, Hari Bathini a écrit :

Quoted 12 lines

Use the newly introduced patch_instructions() that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.

Without this patch (on a POWER9 lpar):

   # time modprobe test_bpf
   real    2m59.681s
   user    0m0.000s
   sys     1m44.160s
   #
Quoted 7 lines

With this patch (on a POWER9 lpar):

   # time modprobe test_bpf
   real    0m5.013s
   user    0m0.000s
   sys     0m4.216s
   #

Right, significant improvement. Forget by comment to patch 1, I should have read the series up to the end. Just wondering why you don't just put patch 4 up front ?

Christophe

Quoted 68 lines
---
  arch/powerpc/net/bpf_jit_comp.c | 30 ++++--------------------------
  1 file changed, 4 insertions(+), 26 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +4 -26
index c60d7570e05d..1e5000d18321 100644
@@ -26,28 +26,6 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)
  	memset32(area, BREAKPOINT_INSTRUCTION, size / 4);
  }
 
-/*
- * Patch 'len' bytes of instructions from opcode to addr, one instruction
- * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise.
- */
-static void *bpf_patch_instructions(void *addr, void *opcode, size_t len, bool fill_insn)
-{
-	while (len > 0) {
-		ppc_inst_t insn = ppc_inst_read(opcode);
-		int ilen = ppc_inst_len(insn);
-
-		if (patch_instruction(addr, insn))
-			return ERR_PTR(-EINVAL);
-
-		len -= ilen;
-		addr = addr + ilen;
-		if (!fill_insn)
-			opcode = opcode + ilen;
-	}
-
-	return addr;
-}
-
  int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, long exit_addr)
  {
  	if (!exit_addr || is_offset_in_branch_range(exit_addr - (ctx->idx * 4))) {
@@ -330,16 +308,16 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int pass
 
  void *bpf_arch_text_copy(void *dst, void *src, size_t len)
  {
-	void *ret;
+	int err;
 
  	if (WARN_ON_ONCE(core_kernel_text((unsigned long)dst)))
  		return ERR_PTR(-EINVAL);
 
  	mutex_lock(&text_mutex);
-	ret = bpf_patch_instructions(dst, src, len, false);
+	err = patch_instructions(dst, src, len, false);
  	mutex_unlock(&text_mutex);
 
-	return ret;
+	return err ? ERR_PTR(err) : dst;
  }
 
  int bpf_arch_text_invalidate(void *dst, size_t len)
@@ -351,7 +329,7 @@ int bpf_arch_text_invalidate(void *dst, size_t len)
  		return -EINVAL;
 
  	mutex_lock(&text_mutex);
-	ret = IS_ERR(bpf_patch_instructions(dst, &insn, len, true));
+	ret = patch_instructions(dst, &insn, len, true);
  	mutex_unlock(&text_mutex);
 
  	return ret;
author maintainer
#10 lore raw

Re: [PATCH v3 5/5] powerpc/bpf: use patch_instructions()

Quoted 23 lines · Christophe Leroy

Le 25/08/2023 à 17:18, Hari Bathini a écrit :

Quoted 12 lines

Use the newly introduced patch_instructions() that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.

Without this patch (on a POWER9 lpar):

    # time modprobe test_bpf
    real    2m59.681s
    user    0m0.000s
    sys     1m44.160s
    #
Quoted 7 lines

With this patch (on a POWER9 lpar):

    # time modprobe test_bpf
    real    0m5.013s
    user    0m0.000s
    sys     0m4.216s
    #
Quoted 3 lines

Right, significant improvement. Forget by comment to patch 1, I should have read the series up to the end. Just wondering why you don't just put patch 4 up front ?

I wanted to remove the dependency for bpf_prog_pack enablement patches with this improvement, just in case..

- Hari