powerpc/bpf: use BPF prog pack allocator
Subsystems (3)
Hari Bathini
maintainercover powerpc/bpf: use BPF prog pack allocator
Most BPF programs are small, but they consume a page each. For systems with busy traffic and many BPF programs, this may also add significant pressure on instruction TLB. High iTLB pressure usually slows down the whole system causing visible performance degradation for production workloads.
bpf_prog_pack, a customized allocator that packs multiple bpf programs into preallocated memory chunks, was proposed [1] to address it. This series extends this support on powerpc.
Patches 1 & 2 add the arch specific functions needed to support this feature. Patch 3 enables the support for powerpc and ensures cleanup is handled gracefully. Patch 4 introduces patch_instructions() that optimizes some calls while patching more than one instruction. Patch 5 leverages this new function to improve time taken for JIT'ing BPF programs.
Note that the first 3 patches are sufficient to enable the support for bpf_prog_pack on powerpc. Patches 4 & 5 are to improve the JIT compilation time of BPF programs on powerpc.
Thanks to Christophe and Song for reviewing v2 [2].
Changes in v3:
* Fixed segfault issue observed on ppc32 due to inaccurate offset calculation for branching.
* Tried to minimize the performance impact for patch_instruction() with the introduction of patch_instructions().
* Corrected uses of u32* vs ppc_instr_t.
* Moved the change that introduces patch_instructions() to after enabling bpf_prog_pack support.
* Added few comments to improve code readability.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Hari Bathini (5):
powerpc/bpf: implement bpf_arch_text_copy
powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free] powerpc/code-patching: introduce patch_instructions() powerpc/bpf: use patch_instructions()
arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/lib/code-patching.c | 94 ++++++++++++--- arch/powerpc/net/bpf_jit.h | 12 +- arch/powerpc/net/bpf_jit_comp.c | 138 ++++++++++++++++++----- arch/powerpc/net/bpf_jit_comp32.c | 13 ++- arch/powerpc/net/bpf_jit_comp64.c | 10 +- 6 files changed, 205 insertions(+), 63 deletions(-)
Signature · 1 line
2.41.0
Hari Bathini
maintainerPATCH v3 1/5 powerpc/bpf: implement bpf_arch_text_copy
bpf_arch_text_copy is used to dump JITed binary to RX page, allowing multiple BPF programs to share the same page. Use patch_instruction() to implement it.
--- arch/powerpc/net/bpf_jit_comp.c | 40 ++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-)
arch/powerpc/net/bpf_jit_comp.c +39 -1
@@ -13,9 +13,12 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> -#include <asm/kprobes.h> +#include <linux/memory.h> #include <linux/bpf.h> +#include <asm/kprobes.h> +#include <asm/code-patching.h> +#include "bpf_jit.h" static void bpf_jit_fill_ill_insns (void *area,unsigned int size)
@@ -23,6 +26,27 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)memset32 (area,BREAKPOINT_INSTRUCTION , size /4 ); } +/* +* Patch 'len' bytes of instructions from opcode to addr, one instruction +* at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. +*/ +static void *bpf_patch_instructions (void *addr,void *opcode,size_t len) +{ +while (len >0 ) { +ppc_inst_t insn =ppc_inst_read (opcode); +int ilen =ppc_inst_len (insn); + +if (patch_instruction (addr, insn)) +return ERR_PTR (-EINVAL ); + + len -= ilen; + addr = addr + ilen; + opcode = opcode + ilen; + } + +return addr; +} +int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr) {if (!exit_addr ||is_offset_in_branch_range (exit_addr - (ctx->idx *4 ))) {
@@ -274,3 +298,17 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct code ctx->exentry_idx++;return 0 ; } + +void *bpf_arch_text_copy (void *dst,void *src,size_t len) +{ +void *ret; + +if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst))) +return ERR_PTR (-EINVAL ); + +mutex_lock (&text_mutex); + ret =bpf_patch_instructions (dst, src, len); +mutex_unlock (&text_mutex); + +return ret; +}
Signature · 1 line
2.41.0
Re: [PATCH v3 1/5] powerpc/bpf: implement bpf_arch_text_copy
Le 25/08/2023 à 17:18, Hari Bathini a écrit :
Quoted 3 lines
bpf_arch_text_copyis used to dump JITed binary to RX page, allowing multiple BPF programs to share the same page. Usepatch_instruction()to implement it.
By using patch_instruction() for doing that you are mapping and unmapping the same page for each instruction you copy. You should implement something to map a page, copy all instruction going into that page, unmap the page and map next page, etc ....
And then perform the icache flush at the end.
Quoted 70 lines
Signed-off-by Hari Bathini <[email protected]>--- arch/powerpc/net/bpf_jit_comp.c | 40 ++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-)arch/powerpc/net/bpf_jit_comp.c +39
-1@@ -13,9 +13,12 @@#include <linux/netdevice.h> #include <linux/filter.h> #include <linux/if_vlan.h> -#include <asm/kprobes.h> +#include <linux/memory.h> #include <linux/bpf.h> +#include <asm/kprobes.h> +#include <asm/code-patching.h> +#include "bpf_jit.h" static void bpf_jit_fill_ill_insns (void *area,unsigned int size)@@ -23,6 +26,27 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)memset32 (area,BREAKPOINT_INSTRUCTION , size /4 ); } +/* +* Patch 'len' bytes of instructions from opcode to addr, one instruction +* at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. +*/ +static void *bpf_patch_instructions (void *addr,void *opcode,size_t len) +{ +while (len >0 ) { +ppc_inst_t insn =ppc_inst_read (opcode); +int ilen =ppc_inst_len (insn); + +if (patch_instruction (addr, insn)) +return ERR_PTR (-EINVAL ); + + len -= ilen; + addr = addr + ilen; + opcode = opcode + ilen; + } + +return addr; +} +int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr) {if (!exit_addr ||is_offset_in_branch_range (exit_addr - (ctx->idx *4 ))) {@@ -274,3 +298,17 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, int pass, struct code ctx->exentry_idx++;return 0 ; } + +void *bpf_arch_text_copy (void *dst,void *src,size_t len) +{ +void *ret; + +if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst))) +return ERR_PTR (-EINVAL ); + +mutex_lock (&text_mutex); + ret =bpf_patch_instructions (dst, src, len); +mutex_unlock (&text_mutex); + +return ret; +}
Hari Bathini
maintainerPATCH v3 2/5 powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack
In reply to Hari BathiniImplement bpf_arch_text_invalidate and use it to fill unused part of the bpf_prog_pack with trap instructions when a BPF program is freed.
--- arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +19 -3
@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)* Patch 'len' bytes of instructions from opcode to addr, one instruction * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. */ -static void *bpf_patch_instructions (void *addr,void *opcode,size_t len) +static void *bpf_patch_instructions (void *addr,void *opcode,size_t len,bool fill_insn) {while (len >0 ) {ppc_inst_t insn =ppc_inst_read (opcode);
@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len) len -= ilen; addr = addr + ilen; - opcode = opcode + ilen; +if (!fill_insn) + opcode = opcode + ilen; }return addr;
@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)return ERR_PTR (-EINVAL );mutex_lock (&text_mutex); - ret =bpf_patch_instructions (dst, src, len); + ret =bpf_patch_instructions (dst, src, len, false); +mutex_unlock (&text_mutex); + +return ret; +} + +int bpf_arch_text_invalidate (void *dst,size_t len) +{ +u32 insn =BREAKPOINT_INSTRUCTION ; +int ret; + +if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst))) +return -EINVAL ; + +mutex_lock (&text_mutex); + ret =IS_ERR (bpf_patch_instructions (dst, &insn, len, true));mutex_unlock (&text_mutex);return ret;
Signature · 1 line
2.41.0
Re: [PATCH v3 2/5] powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack
Le 25/08/2023 à 17:18, Hari Bathini a écrit :
Quoted 13 lines
Implement
bpf_arch_text_invalidateand use it to fill unused part of thebpf_prog_packwith trap instructions when a BPF program is freed.Signed-off-by Hari Bathini <[email protected]>--- arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)arch/powerpc/net/bpf_jit_comp.c +0
-0
arch/powerpc/net/bpf_jit_comp.c static void bpf_jit_fill_ill_insns(void *area, unsigned int size) · line 33
Quoted 5 lines
arch/powerpc/net/bpf_jit_comp.c +1
-1@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)* Patch 'len' bytes of instructions from opcode to addr, one instruction * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. */ -static void *bpf_patch_instructions (void *addr,void *opcode,size_t len) +static void *bpf_patch_instructions (void *addr,void *opcode,size_t len,bool fill_insn)
It's a pitty that you have to modify in patch 2 a function you have added in patch 1 of the same series. Can't you have it right from the begining ?
Quoted 24 lines
arch/powerpc/net/bpf_jit_comp.c +8
-2{while (len >0 ) {ppc_inst_t insn =ppc_inst_read (opcode);@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len) len -= ilen; addr = addr + ilen; - opcode = opcode + ilen; +if (!fill_insn) + opcode = opcode + ilen; }return addr;@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)return ERR_PTR (-EINVAL );mutex_lock (&text_mutex); - ret =bpf_patch_instructions (dst, src, len); + ret =bpf_patch_instructions (dst, src, len, false); +mutex_unlock (&text_mutex); + +return ret; +} +
arch/powerpc/net/bpf_jit_comp.c void *bpf_arch_text_copy(void *dst, void *src, size_t len) · line 326
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +10
-0+int bpf_arch_text_invalidate (void *dst,size_t len) +{ +u32 insn =BREAKPOINT_INSTRUCTION ; +int ret; + +if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst))) +return -EINVAL ; + +mutex_lock (&text_mutex); + ret =IS_ERR (bpf_patch_instructions (dst, &insn, len, true));
Why IS_ERR ?
As far as I understand from the weak definition in kernel/bpf/core.c, this function is supposed to return an error, not a bool.
Quoted 3 lines
arch/powerpc/net/bpf_jit_comp.c +0
-0mutex_unlock (&text_mutex);return ret;
Hari Bathini
maintainerRe: [PATCH v3 2/5] powerpc/bpf: implement bpf_arch_text_invalidate for bpf_prog_pack
Quoted 63 lines · Christophe Leroy
Le 25/08/2023 à 17:18, Hari Bathini a écrit :
Quoted 13 lines
Implement
bpf_arch_text_invalidateand use it to fill unused part of thebpf_prog_packwith trap instructions when a BPF program is freed.Signed-off-by Hari Bathini <[email protected]>--- arch/powerpc/net/bpf_jit_comp.c | 22 +++++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-)arch/powerpc/net/bpf_jit_comp.c +0
-0arch/powerpc/net/bpf_jit_comp.c static void bpf_jit_fill_ill_insns(void *area, unsigned int size) · line 33
Quoted 5 lines
arch/powerpc/net/bpf_jit_comp.c +1
-1@@ -30,7 +30,7 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)* Patch 'len' bytes of instructions from opcode to addr, one instruction * at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. */ -static void *bpf_patch_instructions (void *addr,void *opcode,size_t len) +static void *bpf_patch_instructions (void *addr,void *opcode,size_t len,bool fill_insn)It's a pitty that you have to modify in patch 2 a function you have added in patch 1 of the same series. Can't you have it right from the begining ?
Quoted 24 lines
arch/powerpc/net/bpf_jit_comp.c +8
-2{while (len >0 ) {ppc_inst_t insn =ppc_inst_read (opcode);@@ -41,7 +41,8 @@ static void *bpf_patch_instructions(void *addr, void *opcode, size_t len) len -= ilen; addr = addr + ilen; - opcode = opcode + ilen; +if (!fill_insn) + opcode = opcode + ilen; }return addr;@@ -307,7 +308,22 @@ void *bpf_arch_text_copy(void *dst, void *src, size_t len)return ERR_PTR (-EINVAL );mutex_lock (&text_mutex); - ret =bpf_patch_instructions (dst, src, len); + ret =bpf_patch_instructions (dst, src, len, false); +mutex_unlock (&text_mutex); + +return ret; +} +arch/powerpc/net/bpf_jit_comp.c void *bpf_arch_text_copy(void *dst, void *src, size_t len) · line 326
Quoted 10 lines
arch/powerpc/net/bpf_jit_comp.c +10
-0+int bpf_arch_text_invalidate (void *dst,size_t len) +{ +u32 insn =BREAKPOINT_INSTRUCTION ; +int ret; + +if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst))) +return -EINVAL ; + +mutex_lock (&text_mutex); + ret =IS_ERR (bpf_patch_instructions (dst, &insn, len, true));Why
IS_ERR?
Quoted 2 lines
As far as I understand from the weak definition in
kernel/bpf/core.c, this function is supposed to return an error, not a bool.
My bad! Will fix that in the next revision.
- Hari
Hari Bathini
maintainerPATCH v3 3/5 powerpc/bpf: use bpf_jit_binary_pack_[alloc|finalize|free]
In reply to Hari BathiniUse bpf_jit_binary_pack_alloc in powerpc jit. The jit engine first writes the program to the rw buffer. When the jit is done, the program is copied to the final location with bpf_jit_binary_pack_finalize. With multiple jit_subprogs, bpf_jit_free is called on some subprograms that haven't got bpf_jit_binary_pack_finalize() yet. Implement custom bpf_jit_free() like in commit 1d5f82d9dd47 ("bpf, x86: fix freeing of not-finalized bpf_prog_pack") to call bpf_jit_binary_pack_finalize(), if necessary. While here, correct the misnomer powerpc64_jit_data to powerpc_jit_data as it is meant for both ppc32 and ppc64.
--- arch/powerpc/net/bpf_jit.h | 12 ++-- arch/powerpc/net/bpf_jit_comp.c | 110 ++++++++++++++++++++++-------- arch/powerpc/net/bpf_jit_comp32.c | 13 ++-- arch/powerpc/net/bpf_jit_comp64.c | 10 +-- 4 files changed, 98 insertions(+), 47 deletions(-)
arch/powerpc/net/bpf_jit.h +5 -7
@@ -36,9 +36,6 @@EMIT (PPC_RAW_BRANCH (offset)); \ }while (0 ) -/* bl (unconditional 'branch' with link) */ -#define PPC_BL (dest) EMIT(PPC_RAW_BL((dest) - (unsigned long)(image + ctx->idx))) - /* "cond" here covers BO:BI fields. */#define PPC_BCC_SHORT (cond, dest) \ do { \
@@ -169,16 +166,17 @@ static inline void bpf_clear_seen_register(struct codegen_context *ctx, int i) }void bpf_jit_init_reg_mapping (struct codegen_context *ctx); -int bpf_jit_emit_func_call_rel (u32 *image,struct codegen_context *ctx,u64 func); -int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,struct codegen_context *ctx, +int bpf_jit_emit_func_call_rel (u32 *image,u32 *fimage,struct codegen_context *ctx,u64 func); +int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,u32 *fimage,struct codegen_context *ctx,u32 *addrs,int pass,bool extra_pass);void bpf_jit_build_prologue (u32 *image,struct codegen_context *ctx);void bpf_jit_build_epilogue (u32 *image,struct codegen_context *ctx);void bpf_jit_realloc_regs (struct codegen_context *ctx);int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr); -int bpf_add_extable_entry (struct bpf_prog *fp,u32 *image,int pass,struct codegen_context *ctx, -int insn_idx,int jmp_off,int dst_reg); +int bpf_add_extable_entry (struct bpf_prog *fp,u32 *image,u32 *fimage,int pass, +struct codegen_context *ctx,int insn_idx, +int jmp_off,int dst_reg);#endif
arch/powerpc/net/bpf_jit_comp.c +81 -29
220 lines not on this page · raw message
arch/powerpc/net/bpf_jit_comp32.c +7 -6
@@ -200,12 +200,13 @@ void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)EMIT (PPC_RAW_BLR ()); } -int bpf_jit_emit_func_call_rel (u32 *image,struct codegen_context *ctx,u64 func) +/* Relative offset needs to be calculated based on final image location */ +int bpf_jit_emit_func_call_rel (u32 *image,u32 *fimage,struct codegen_context *ctx,u64 func) { -s32 rel = (s32 )func - (s32)(image + ctx->idx); +s32 rel = (s32 )func - (s32)(fimage + ctx->idx);if (image && rel <0x2000000 && rel >=-0x2000000 ) { -PPC_BL (func); +EMIT (PPC_RAW_BL (rel)); }else {/* Load function address into r0 */ EMIT (PPC_RAW_LIS (_R0,IMM_H (func)));
@@ -278,7 +279,7 @@ static int bpf_jit_emit_tail_call(u32 *image, struct codegen_context *ctx, u32 o }/* Assemble the body code between the prologue & epilogue */ -int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,struct codegen_context *ctx, +int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,u32 *fimage,struct codegen_context *ctx,u32 *addrs,int pass,bool extra_pass) {const struct bpf_insn *insn = fp->insnsi;
@@ -997,7 +998,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * jmp_off +=4 ; } - ret =bpf_add_extable_entry (fp, image, pass, ctx, insn_idx, + ret =bpf_add_extable_entry (fp, image, fimage, pass, ctx, insn_idx, jmp_off, dst_reg);if (ret)return ret;
@@ -1053,7 +1054,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *EMIT (PPC_RAW_STW (bpf_to_ppc (BPF_REG_5 ), _R1,12 )); } - ret =bpf_jit_emit_func_call_rel (image, ctx, func_addr); + ret =bpf_jit_emit_func_call_rel (image, fimage, ctx, func_addr);if (ret)return ret;
arch/powerpc/net/bpf_jit_comp64.c +5 -5
@@ -240,7 +240,7 @@ static int bpf_jit_emit_func_call_hlp(u32 *image, struct codegen_context *ctx, ureturn 0 ; } -int bpf_jit_emit_func_call_rel (u32 *image,struct codegen_context *ctx,u64 func) +int bpf_jit_emit_func_call_rel (u32 *image,u32 *fimage,struct codegen_context *ctx,u64 func) {unsigned int i, ctx_idx = ctx->idx;
@@ -361,7 +361,7 @@ asm ( );/* Assemble the body code between the prologue & epilogue */ -int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,struct codegen_context *ctx, +int bpf_jit_build_body (struct bpf_prog *fp,u32 *image,u32 *fimage,struct codegen_context *ctx,u32 *addrs,int pass,bool extra_pass) {enum stf_barrier_type stf_barrier =stf_barrier_type_get ();
@@ -940,8 +940,8 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * addrs[++i] = ctx->idx *4 ;if (BPF_MODE (code) ==BPF_PROBE_MEM ) { - ret =bpf_add_extable_entry (fp, image, pass, ctx, ctx->idx -1 , -4 , dst_reg); + ret =bpf_add_extable_entry (fp, image, fimage, pass, ctx, + ctx->idx -1 ,4 , dst_reg);if (ret)return ret; }
@@ -995,7 +995,7 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context *if (func_addr_fixed) ret =bpf_jit_emit_func_call_hlp (image, ctx, func_addr);else - ret =bpf_jit_emit_func_call_rel (image, ctx, func_addr); + ret =bpf_jit_emit_func_call_rel (image, fimage, ctx, func_addr);if (ret)return ret;
Signature · 1 line
2.41.0
Hari Bathini
maintainerPATCH v3 4/5 powerpc/code-patching: introduce patch_instructions()
In reply to Hari Bathinipatch_instruction() entails setting up pte, patching the instruction, clearing the pte and flushing the tlb. If multiple instructions need to be patched, every instruction would have to go through the above drill unnecessarily. Instead, introduce function patch_instructions() that sets up the pte, clears the pte and flushes the tlb only once per page range of instructions to be patched. This adds a slight overhead to patch_instruction() call while improving the patching time for scenarios where more than one instruction needs to be patched.
--- arch/powerpc/include/asm/code-patching.h | 1 + arch/powerpc/lib/code-patching.c | 94 ++++++++++++++++++++---- 2 files changed, 80 insertions(+), 15 deletions(-)
arch/powerpc/include/asm/code-patching.h +1 -0
@@ -74,6 +74,7 @@ int create_cond_branch(ppc_inst_t *instr, const u32 *addr,int patch_branch (u32 *addr,unsigned long target,int flags);int patch_instruction (u32 *addr,ppc_inst_t instr);int raw_patch_instruction (u32 *addr,ppc_inst_t instr); +int patch_instructions (void *addr,void *code,size_t len,bool fill_insn);static inline unsigned long patch_site_addr (s32 *site) {
arch/powerpc/lib/code-patching.c +79 -15
148 lines not on this page · raw message
Signature · 1 line
2.41.0
Hari Bathini
maintainerPATCH v3 5/5 powerpc/bpf: use patch_instructions()
In reply to Hari BathiniUse the newly introduced patch_instructions() that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.
Without this patch (on a POWER9 lpar):
# time modprobe test_bpf real 2m59.681s user 0m0.000s sys 1m44.160s #
With this patch (on a POWER9 lpar):
# time modprobe test_bpf real 0m5.013s user 0m0.000s sys 0m4.216s #
--- arch/powerpc/net/bpf_jit_comp.c | 30 ++++-------------------------- 1 file changed, 4 insertions(+), 26 deletions(-)
arch/powerpc/net/bpf_jit_comp.c +4 -26
@@ -26,28 +26,6 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)memset32 (area,BREAKPOINT_INSTRUCTION , size /4 ); } -/* -* Patch 'len' bytes of instructions from opcode to addr, one instruction -* at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. -*/ -static void *bpf_patch_instructions (void *addr,void *opcode,size_t len,bool fill_insn) -{ -while (len >0 ) { -ppc_inst_t insn =ppc_inst_read (opcode); -int ilen =ppc_inst_len (insn); - -if (patch_instruction (addr, insn)) -return ERR_PTR (-EINVAL ); - - len -= ilen; - addr = addr + ilen; -if (!fill_insn) - opcode = opcode + ilen; - } - -return addr; -} -int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr) {if (!exit_addr ||is_offset_in_branch_range (exit_addr - (ctx->idx *4 ))) {
@@ -330,16 +308,16 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int passvoid *bpf_arch_text_copy (void *dst,void *src,size_t len) { -void *ret; +int err;if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst)))return ERR_PTR (-EINVAL );mutex_lock (&text_mutex); - ret =bpf_patch_instructions (dst, src, len, false); + err =patch_instructions (dst, src, len, false);mutex_unlock (&text_mutex); -return ret; +return err ?ERR_PTR (err) : dst; }int bpf_arch_text_invalidate (void *dst,size_t len)
@@ -351,7 +329,7 @@ int bpf_arch_text_invalidate(void *dst, size_t len)return -EINVAL ;mutex_lock (&text_mutex); - ret =IS_ERR (bpf_patch_instructions (dst, &insn, len, true)); + ret =patch_instructions (dst, &insn, len, true);mutex_unlock (&text_mutex);return ret;
Signature · 1 line
2.41.0
Re: [PATCH v3 5/5] powerpc/bpf: use patch_instructions()
Le 25/08/2023 à 17:18, Hari Bathini a écrit :
Quoted 12 lines
Use the newly introduced
patch_instructions()that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.Without this patch (on a POWER9 lpar):
# time modprobe test_bpf real 2m59.681s user 0m0.000s sys 1m44.160s #
Quoted 7 lines
With this patch (on a POWER9 lpar):
# time modprobe test_bpf real 0m5.013s user 0m0.000s sys 0m4.216s #
Right, significant improvement. Forget by comment to patch 1, I should have read the series up to the end. Just wondering why you don't just put patch 4 up front ?
Christophe
Quoted 68 lines
Signed-off-by Hari Bathini <[email protected]>--- arch/powerpc/net/bpf_jit_comp.c | 30 ++++-------------------------- 1 file changed, 4 insertions(+), 26 deletions(-)arch/powerpc/net/bpf_jit_comp.c +4
-26@@ -26,28 +26,6 @@ static void bpf_jit_fill_ill_insns(void *area, unsigned int size)memset32 (area,BREAKPOINT_INSTRUCTION , size /4 ); } -/* -* Patch 'len' bytes of instructions from opcode to addr, one instruction -* at a time. Returns addr on success. ERR_PTR(-EINVAL), otherwise. -*/ -static void *bpf_patch_instructions (void *addr,void *opcode,size_t len,bool fill_insn) -{ -while (len >0 ) { -ppc_inst_t insn =ppc_inst_read (opcode); -int ilen =ppc_inst_len (insn); - -if (patch_instruction (addr, insn)) -return ERR_PTR (-EINVAL ); - - len -= ilen; - addr = addr + ilen; -if (!fill_insn) - opcode = opcode + ilen; - } - -return addr; -} -int bpf_jit_emit_exit_insn (u32 *image,struct codegen_context *ctx,int tmp_reg,long exit_addr) {if (!exit_addr ||is_offset_in_branch_range (exit_addr - (ctx->idx *4 ))) {@@ -330,16 +308,16 @@ int bpf_add_extable_entry(struct bpf_prog *fp, u32 *image, u32 *fimage, int passvoid *bpf_arch_text_copy (void *dst,void *src,size_t len) { -void *ret; +int err;if (WARN_ON_ONCE (core_kernel_text ((unsigned long )dst)))return ERR_PTR (-EINVAL );mutex_lock (&text_mutex); - ret =bpf_patch_instructions (dst, src, len, false); + err =patch_instructions (dst, src, len, false);mutex_unlock (&text_mutex); -return ret; +return err ?ERR_PTR (err) : dst; }int bpf_arch_text_invalidate (void *dst,size_t len)@@ -351,7 +329,7 @@ int bpf_arch_text_invalidate(void *dst, size_t len)return -EINVAL ;mutex_lock (&text_mutex); - ret =IS_ERR (bpf_patch_instructions (dst, &insn, len, true)); + ret =patch_instructions (dst, &insn, len, true);mutex_unlock (&text_mutex);return ret;
Hari Bathini
maintainerRe: [PATCH v3 5/5] powerpc/bpf: use patch_instructions()
Quoted 23 lines · Christophe Leroy
Le 25/08/2023 à 17:18, Hari Bathini a écrit :
Quoted 12 lines
Use the newly introduced
patch_instructions()that handles patching multiple instructions with one call. This improves speed of exectution for JIT'ing bpf programs.Without this patch (on a POWER9 lpar):
# time modprobe test_bpf real 2m59.681s user 0m0.000s sys 1m44.160s #Quoted 7 lines
With this patch (on a POWER9 lpar):
# time modprobe test_bpf real 0m5.013s user 0m0.000s sys 0m4.216s #
Quoted 3 lines
Right, significant improvement. Forget by comment to patch 1, I should have read the series up to the end. Just wondering why you don't just put patch 4 up front ?
I wanted to remove the dependency for bpf_prog_pack enablement patches with this improvement, just in case..
- Hari