bpf: Yet another approach to fix the BPF dispatcher thing
Subsystems (6)
cover bpf: Yet another approach to fix the BPF dispatcher thing
Hi!
Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.
This is another approach -- using static_call() to rewrite the dispatcher function. I've compile tested this on:
x86_64 (inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)
A previous version was tested and found working by Bjorn.
It is split in two patches; first reverting the current approach and then introducing the new for ease of review.
PATCH 1/2 bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop")
Because __attribute__((patchable_function_entry)) is only available since GCC-8 this solution fails to build on the minimum required GCC version.
Undo these changes so we might try again -- without cluttering up the patches with too many changes.
This is an almost complete revert of:
dbe69b299884 ("bpf: Fix dispatcher patchable function entry to 5 bytes nop") ceea991a019c ("bpf: Move bpf_dispatcher function out of ftrace locations")
(notably the arch/x86/Kconfig hunk is kept).
--- arch/x86/net/bpf_jit_comp.c | 13 ------------- include/linux/bpf.h | 21 +-------------------- kernel/bpf/dispatcher.c | 6 ------ 3 files changed, 1 insertion(+), 39 deletions(-)
arch/x86/net/bpf_jit_comp.c +0 -13
@@ -11,7 +11,6 @@#include <linux/bpf.h> #include <linux/memory.h> #include <linux/sort.h> -#include <linux/init.h> #include <asm/extable.h> #include <asm/set_memory.h> #include <asm/nospec-branch.h>
@@ -389,18 +388,6 @@ static int __bpf_arch_text_poke(void *ipreturn ret; } -int __init bpf_arch_init_dispatcher_early (void *ip) -{ -const u8 *nop_insn = x86_nops[5 ]; - -if (is_endbr (*(u32 *)ip)) - ip +=ENDBR_INSN_SIZE ; - -if (memcmp (ip, nop_insn,X86_PATCH_SIZE )) -text_poke_early (ip, nop_insn,X86_PATCH_SIZE ); -return 0 ; -} -int bpf_arch_text_poke (void *ip,enum bpf_text_poke_type t,void *old_addr,void *new_addr) {
include/linux/bpf.h +1 -20
@@ -27,7 +27,6 @@#include <linux/bpfptr.h> #include <linux/btf.h> #include <linux/rcupdate_trace.h> -#include <linux/init.h> struct bpf_verifier_env ;struct bpf_verifier_log ;
@@ -971,8 +970,6 @@ struct bpf_trampoline *bpf_trampoline_ge struct bpf_attach_target_info *tgt_info);void bpf_trampoline_put (struct bpf_trampoline *tr);int arch_prepare_bpf_dispatcher (void *image,void *buf,s64 *funcs,int num_funcs); -int __init bpf_arch_init_dispatcher_early (void *ip); -#define BPF_DISPATCHER_INIT (_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \
@@ -986,21 +983,7 @@ int __init bpf_arch_init_dispatcher_earl }, \ } -#define BPF_DISPATCHER_INIT_CALL (_name) \ - static int __init _name##_init(void) \ - { \ - return bpf_arch_init_dispatcher_early(_name##_func); \ - } \ - early_initcall(_name##_init) - -#ifdef CONFIG_X86_64 -#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) -#else -#define BPF_DISPATCHER_ATTRIBUTES -#endif -#define DEFINE_BPF_DISPATCHER (name) \ - notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \
@@ -1010,9 +993,7 @@ int __init bpf_arch_init_dispatcher_earl } \ EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \ struct bpf_dispatcher bpf_dispatcher_##name = \ - BPF_DISPATCHER_INIT(bpf_dispatcher_##name); \ - BPF_DISPATCHER_INIT_CALL(bpf_dispatcher_##name); - + BPF_DISPATCHER_INIT(bpf_dispatcher_##name);#define DECLARE_BPF_DISPATCHER (name) \ unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \
kernel/bpf/dispatcher.c +0 -6
@@ -4,7 +4,6 @@#include <linux/hash.h> #include <linux/bpf.h> #include <linux/filter.h> -#include <linux/init.h> /* The BPF dispatcher is a multiway branch code generator. The * dispatcher is a mechanism to avoid the performance penalty of an
@@ -90,11 +89,6 @@ int __weak arch_prepare_bpf_dispatcher(v {return -ENOTSUPP ; } - -int __weak __init bpf_arch_init_dispatcher_early (void *ip) -{ -return -ENOTSUPP ; -}static int bpf_dispatcher_prepare (struct bpf_dispatcher *d,void *image,void *buf) {
PATCH 2/2 bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
In reply to Peter ZijlstraThe dispatcher function is currently abusing the ftrace __fentry__ call location for its own purposes -- this obviously gives trouble when the dispatcher and ftrace are both in use.
A previous solution tried using __attribute__((patchable_function_entry())) which works, except it is GCC-8+ only, breaking the build on the earlier still supported compilers. Instead use static_call() -- which has its own annotations and does not conflict with ftrace -- to rewrite the dispatch function.
By using: return static_call()(ctx, insni, bpf_func) you get a perfect forwarding tail call as function body (iow a single jmp instruction). By having the default static_call() target be bpf_dispatcher_nop_func() it retains the default behaviour (an indirect call to the argument function). Only
once a dispatcher program is attached is the target rewritten to directly call the JIT'ed image.
--- include/linux/bpf.h | 39 ++++++++++++++++++++++++++++++++++++++- kernel/bpf/dispatcher.c | 22 ++++++++-------------- 2 files changed, 46 insertions(+), 15 deletions(-)
include/linux/bpf.h +38 -1
@@ -27,6 +27,7 @@#include <linux/bpfptr.h> #include <linux/btf.h> #include <linux/rcupdate_trace.h> +#include <linux/static_call.h> struct bpf_verifier_env ;struct bpf_verifier_log ;
@@ -953,6 +954,10 @@ struct bpf_dispatcher {void *rw_image;u32 image_off;struct bpf_ksym ksym; +#ifdef CONFIG_HAVE_STATIC_CALL +struct static_call_key *sc_key; +void *sc_tramp; +#endif }; static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
@@ -970,6 +975,34 @@ struct bpf_trampoline *bpf_trampoline_ge struct bpf_attach_target_info *tgt_info);void bpf_trampoline_put (struct bpf_trampoline *tr);int arch_prepare_bpf_dispatcher (void *image,void *buf,s64 *funcs,int num_funcs); + +/* +* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn +* indirection with a direct call to the bpf program. If the architecture does +* not have STATIC_CALL, avoid a double-indirection. +*/ +#ifdef CONFIG_HAVE_STATIC_CALL + +#define __BPF_DISPATCHER_SC_INIT (_name) \ + .sc_key = &STATIC_CALL_KEY(_name), \ + .sc_tramp = STATIC_CALL_TRAMP_ADDR(_name), + +#define __BPF_DISPATCHER_SC (name) \ + DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func) + +#define __BPF_DISPATCHER_CALL (name) \ + static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func) + +#define __BPF_DISPATCHER_UPDATE (_d, _new) \ + __static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new)) + +#else +#define __BPF_DISPATCHER_SC_INIT (name) +#define __BPF_DISPATCHER_SC (name) +#define __BPF_DISPATCHER_CALL (name) bpf_func(ctx, insnsi) +#define __BPF_DISPATCHER_UPDATE (_d, _new) +#endif +#define BPF_DISPATCHER_INIT (_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \
@@ -981,25 +1014,29 @@ int arch_prepare_bpf_dispatcher(void *im .name = #_name, \ .lnode = LIST_HEAD_INIT(_name.ksym.lnode), \ }, \ + __BPF_DISPATCHER_SC_INIT(_name##_call) \ }#define DEFINE_BPF_DISPATCHER (name) \ + __BPF_DISPATCHER_SC(name); \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func) \ { \ - return bpf_func(ctx, insnsi); \ + return __BPF_DISPATCHER_CALL(name); \ } \ EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \ struct bpf_dispatcher bpf_dispatcher_##name = \ BPF_DISPATCHER_INIT(bpf_dispatcher_##name); +#define DECLARE_BPF_DISPATCHER (name) \ unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func); \ extern struct bpf_dispatcher bpf_dispatcher_##name; +#define BPF_DISPATCHER_FUNC (name) bpf_dispatcher_##name##_func#define BPF_DISPATCHER_PTR (name) (&bpf_dispatcher_##name) void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
kernel/bpf/dispatcher.c +8 -14
@@ -4,6 +4,7 @@#include <linux/hash.h> #include <linux/bpf.h> #include <linux/filter.h> +#include <linux/static_call.h> /* The BPF dispatcher is a multiway branch code generator. The * dispatcher is a mechanism to avoid the performance penalty of an
@@ -104,17 +105,11 @@ static int bpf_dispatcher_prepare(structstatic void bpf_dispatcher_update (struct bpf_dispatcher *d,int prev_num_progs) { -void *old, *new, *tmp; -u32 noff; -int err; - -if (!prev_num_progs) { - old =NULL ; - noff =0 ; - }else { - old = d->image + d->image_off; +void *new, *tmp; +u32 noff =0 ; + +if (prev_num_progs) noff = d->image_off ^ (PAGE_SIZE /2 ); - } new = d->num_progs ? d->image + noff :NULL ; tmp = d->num_progs ? d->rw_image + noff :NULL ;
@@ -128,11 +123,10 @@ static void bpf_dispatcher_update(structreturn ; } - err =bpf_arch_text_poke (d->func,BPF_MOD_JUMP , old, new); -if (err || !new) -return ; +__BPF_DISPATCHER_UPDATE (d, new ?: &bpf_dispatcher_nop_func); - d->image_off = noff; +if (new) + d->image_off = noff; } void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
Re: [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
Quoted 3 lines · On Thu, 03 Nov 2022 13:00:14 +0100 Peter Zijlstra <[email protected]> wrote:
The dispatcher function is currently abusing the ftrace
__fentry__call location for its own purposes -- this obviously gives trouble when the dispatcher and ftrace are both in use.
Awesome! Thanks for doing this Peter!
-- Steve
Quoted 10 lines · Peter Zijlstra
Hi!
Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.
This is another approach -- using
static_call()to rewrite the dispatcher function. I've compile tested this on:
Quoted 8 lines
x86_64(inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)A previous version was tested and found working by Bjorn.
It is split in two patches; first reverting the current approach and then introducing the new for ease of review.
Took it for a spin on x86_64/KVM. For the series:
Quoted 10 lines · Peter Zijlstra
Hi!
Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.
This is another approach -- using
static_call()to rewrite the dispatcher function. I've compile tested this on:
Quoted 9 lines
x86_64(inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)A previous version was tested and found working by Bjorn.
It is split in two patches; first reverting the current approach and then introducing the new for ease of review.
thanks,
jirka
This series was applied to bpf/bpf.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <[email protected]>:
Quoted 8 lines · On Thu, 03 Nov 2022 13:00:12 +0100 you wrote:
Hi!
Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.
[...]
Here is the summary with links:
- [1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") (no matching commit)
- [2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)
https://git.kernel.org/bpf/bpf/c/c86df29d11df
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html