Skip to contentSkip to conversation

bpf: Yet another approach to fix the BPF dispatcher thing

PATCH In review Peter Zijlstra 5 participants 7 messages Nov 3, 2022 – Nov 14, 2022

Subsystems (6)
Networking Network drivers x86 BPF +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover bpf: Yet another approach to fix the BPF dispatcher thing

Hi!

Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.

This is another approach -- using static_call() to rewrite the dispatcher function. I've compile tested this on:

x86_64 (inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)

A previous version was tested and found working by Bjorn.

It is split in two patches; first reverting the current approach and then introducing the new for ease of review.

#1 lore raw

PATCH 1/2 bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop")

Because __attribute__((patchable_function_entry)) is only available since GCC-8 this solution fails to build on the minimum required GCC version.

Undo these changes so we might try again -- without cluttering up the patches with too many changes.

This is an almost complete revert of:

dbe69b299884 ("bpf: Fix dispatcher patchable function entry to 5 bytes nop") ceea991a019c ("bpf: Move bpf_dispatcher function out of ftrace locations")

(notably the arch/x86/Kconfig hunk is kept).

---
 arch/x86/net/bpf_jit_comp.c |   13 -------------
 include/linux/bpf.h         |   21 +--------------------
 kernel/bpf/dispatcher.c     |    6 ------
 3 files changed, 1 insertion(+), 39 deletions(-)
arch/x86/net/bpf_jit_comp.c +0 -13
@@ -11,7 +11,6 @@
 #include <linux/bpf.h>
 #include <linux/memory.h>
 #include <linux/sort.h>
-#include <linux/init.h>
 #include <asm/extable.h>
 #include <asm/set_memory.h>
 #include <asm/nospec-branch.h>
@@ -389,18 +388,6 @@ static int __bpf_arch_text_poke(void *ip
 	return ret;
 }
 
-int __init bpf_arch_init_dispatcher_early(void *ip)
-{
-	const u8 *nop_insn = x86_nops[5];
-
-	if (is_endbr(*(u32 *)ip))
-		ip += ENDBR_INSN_SIZE;
-
-	if (memcmp(ip, nop_insn, X86_PATCH_SIZE))
-		text_poke_early(ip, nop_insn, X86_PATCH_SIZE);
-	return 0;
-}
-
 int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t,
 		       void *old_addr, void *new_addr)
 {
include/linux/bpf.h +1 -20
@@ -27,7 +27,6 @@
 #include <linux/bpfptr.h>
 #include <linux/btf.h>
 #include <linux/rcupdate_trace.h>
-#include <linux/init.h>
 
 struct bpf_verifier_env;
 struct bpf_verifier_log;
@@ -971,8 +970,6 @@ struct bpf_trampoline *bpf_trampoline_ge
 					  struct bpf_attach_target_info *tgt_info);
 void bpf_trampoline_put(struct bpf_trampoline *tr);
 int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
-int __init bpf_arch_init_dispatcher_early(void *ip);
-
 #define BPF_DISPATCHER_INIT(_name) {				\
 	.mutex = __MUTEX_INITIALIZER(_name.mutex),		\
 	.func = &_name##_func,					\
@@ -986,21 +983,7 @@ int __init bpf_arch_init_dispatcher_earl
 	},							\
 }
 
-#define BPF_DISPATCHER_INIT_CALL(_name)					\
-	static int __init _name##_init(void)				\
-	{								\
-		return bpf_arch_init_dispatcher_early(_name##_func);	\
-	}								\
-	early_initcall(_name##_init)
-
-#ifdef CONFIG_X86_64
-#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
-#else
-#define BPF_DISPATCHER_ATTRIBUTES
-#endif
-
 #define DEFINE_BPF_DISPATCHER(name)					\
-	notrace BPF_DISPATCHER_ATTRIBUTES				\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
@@ -1010,9 +993,7 @@ int __init bpf_arch_init_dispatcher_earl
 	}								\
 	EXPORT_SYMBOL(bpf_dispatcher_##name##_func);			\
 	struct bpf_dispatcher bpf_dispatcher_##name =			\
-		BPF_DISPATCHER_INIT(bpf_dispatcher_##name);		\
-	BPF_DISPATCHER_INIT_CALL(bpf_dispatcher_##name);
-
+		BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
 #define DECLARE_BPF_DISPATCHER(name)					\
 	unsigned int bpf_dispatcher_##name##_func(			\
 		const void *ctx,					\
kernel/bpf/dispatcher.c +0 -6
@@ -4,7 +4,6 @@
 #include <linux/hash.h>
 #include <linux/bpf.h>
 #include <linux/filter.h>
-#include <linux/init.h>
 
 /* The BPF dispatcher is a multiway branch code generator. The
  * dispatcher is a mechanism to avoid the performance penalty of an
@@ -90,11 +89,6 @@ int __weak arch_prepare_bpf_dispatcher(v
 {
 	return -ENOTSUPP;
 }
-
-int __weak __init bpf_arch_init_dispatcher_early(void *ip)
-{
-	return -ENOTSUPP;
-}
 
 static int bpf_dispatcher_prepare(struct bpf_dispatcher *d, void *image, void *buf)
 {
#2 lore raw

PATCH 2/2 bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)

In reply to Peter Zijlstra

The dispatcher function is currently abusing the ftrace __fentry__ call location for its own purposes -- this obviously gives trouble when the dispatcher and ftrace are both in use.

A previous solution tried using __attribute__((patchable_function_entry())) which works, except it is GCC-8+ only, breaking the build on the earlier still supported compilers. Instead use static_call() -- which has its own annotations and does not conflict with ftrace -- to rewrite the dispatch function.

By using: return static_call()(ctx, insni, bpf_func) you get a perfect forwarding tail call as function body (iow a single jmp instruction). By having the default static_call() target be bpf_dispatcher_nop_func() it retains the default behaviour (an indirect call to the argument function). Only
once a dispatcher program is attached is the target rewritten to directly call the JIT'ed image.

---
 include/linux/bpf.h     |   39 ++++++++++++++++++++++++++++++++++++++-
 kernel/bpf/dispatcher.c |   22 ++++++++--------------
 2 files changed, 46 insertions(+), 15 deletions(-)
include/linux/bpf.h +38 -1
@@ -27,6 +27,7 @@
 #include <linux/bpfptr.h>
 #include <linux/btf.h>
 #include <linux/rcupdate_trace.h>
+#include <linux/static_call.h>
 
 struct bpf_verifier_env;
 struct bpf_verifier_log;
@@ -953,6 +954,10 @@ struct bpf_dispatcher {
 	void *rw_image;
 	u32 image_off;
 	struct bpf_ksym ksym;
+#ifdef CONFIG_HAVE_STATIC_CALL
+	struct static_call_key *sc_key;
+	void *sc_tramp;
+#endif
 };
 
 static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
@@ -970,6 +975,34 @@ struct bpf_trampoline *bpf_trampoline_ge
 					  struct bpf_attach_target_info *tgt_info);
 void bpf_trampoline_put(struct bpf_trampoline *tr);
 int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
+
+/*
+ * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
+ * indirection with a direct call to the bpf program. If the architecture does
+ * not have STATIC_CALL, avoid a double-indirection.
+ */
+#ifdef CONFIG_HAVE_STATIC_CALL
+
+#define __BPF_DISPATCHER_SC_INIT(_name)				\
+	.sc_key = &STATIC_CALL_KEY(_name),			\
+	.sc_tramp = STATIC_CALL_TRAMP_ADDR(_name),
+
+#define __BPF_DISPATCHER_SC(name)				\
+	DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func)
+
+#define __BPF_DISPATCHER_CALL(name)				\
+	static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func)
+
+#define __BPF_DISPATCHER_UPDATE(_d, _new)			\
+	__static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new))
+
+#else
+#define __BPF_DISPATCHER_SC_INIT(name)
+#define __BPF_DISPATCHER_SC(name)
+#define __BPF_DISPATCHER_CALL(name)		bpf_func(ctx, insnsi)
+#define __BPF_DISPATCHER_UPDATE(_d, _new)
+#endif
+
 #define BPF_DISPATCHER_INIT(_name) {				\
 	.mutex = __MUTEX_INITIALIZER(_name.mutex),		\
 	.func = &_name##_func,					\
@@ -981,25 +1014,29 @@ int arch_prepare_bpf_dispatcher(void *im
 		.name  = #_name,				\
 		.lnode = LIST_HEAD_INIT(_name.ksym.lnode),	\
 	},							\
+	__BPF_DISPATCHER_SC_INIT(_name##_call)			\
 }
 
 #define DEFINE_BPF_DISPATCHER(name)					\
+	__BPF_DISPATCHER_SC(name);					\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
 		bpf_func_t bpf_func)					\
 	{								\
-		return bpf_func(ctx, insnsi);				\
+		return __BPF_DISPATCHER_CALL(name);			\
 	}								\
 	EXPORT_SYMBOL(bpf_dispatcher_##name##_func);			\
 	struct bpf_dispatcher bpf_dispatcher_##name =			\
 		BPF_DISPATCHER_INIT(bpf_dispatcher_##name);
+
 #define DECLARE_BPF_DISPATCHER(name)					\
 	unsigned int bpf_dispatcher_##name##_func(			\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
 		bpf_func_t bpf_func);					\
 	extern struct bpf_dispatcher bpf_dispatcher_##name;
+
 #define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func
 #define BPF_DISPATCHER_PTR(name) (&bpf_dispatcher_##name)
 void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
kernel/bpf/dispatcher.c +8 -14
@@ -4,6 +4,7 @@
 #include <linux/hash.h>
 #include <linux/bpf.h>
 #include <linux/filter.h>
+#include <linux/static_call.h>
 
 /* The BPF dispatcher is a multiway branch code generator. The
  * dispatcher is a mechanism to avoid the performance penalty of an
@@ -104,17 +105,11 @@ static int bpf_dispatcher_prepare(struct
 
 static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs)
 {
-	void *old, *new, *tmp;
-	u32 noff;
-	int err;
-
-	if (!prev_num_progs) {
-		old = NULL;
-		noff = 0;
-	} else {
-		old = d->image + d->image_off;
+	void *new, *tmp;
+	u32 noff = 0;
+
+	if (prev_num_progs)
 		noff = d->image_off ^ (PAGE_SIZE / 2);
-	}
 
 	new = d->num_progs ? d->image + noff : NULL;
 	tmp = d->num_progs ? d->rw_image + noff : NULL;
@@ -128,11 +123,10 @@ static void bpf_dispatcher_update(struct
 			return;
 	}
 
-	err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP, old, new);
-	if (err || !new)
-		return;
+	__BPF_DISPATCHER_UPDATE(d, new ?: &bpf_dispatcher_nop_func);
 
-	d->image_off = noff;
+	if (new)
+		d->image_off = noff;
 }
 
 void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,
Acked-by on v1 2/2
#3 lore raw

Re: [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)

Quoted 3 lines · On Thu, 03 Nov 2022 13:00:14 +0100 Peter Zijlstra <[email protected]> wrote:

The dispatcher function is currently abusing the ftrace __fentry__ call location for its own purposes -- this obviously gives trouble when the dispatcher and ftrace are both in use.

Awesome! Thanks for doing this Peter!

-- Steve

Acked-by Tested-by on the v1 series
#4 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

Hi!

Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.

This is another approach -- using static_call() to rewrite the dispatcher function. I've compile tested this on:

Quoted 8 lines

x86_64 (inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)

A previous version was tested and found working by Bjorn.

It is split in two patches; first reverting the current approach and then introducing the new for ease of review.

Took it for a spin on x86_64/KVM. For the series:

Acked-by Tested-by on the v1 series
#5 lore raw
In reply to Peter Zijlstra
Quoted 10 lines · Peter Zijlstra

Hi!

Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.

This is another approach -- using static_call() to rewrite the dispatcher function. I've compile tested this on:

Quoted 9 lines

x86_64 (inline static-call support)
i386 (out-of-line static-call support)
aargh64 (no static-call support)

A previous version was tested and found working by Bjorn.

It is split in two patches; first reverting the current approach and then introducing the new for ease of review.

thanks,
jirka

#6 lore raw
In reply to Peter Zijlstra
This series was applied to bpf/bpf.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This series was applied to bpf/bpf.git (master)
by Daniel Borkmann <[email protected]>:

Quoted 8 lines · On Thu, 03 Nov 2022 13:00:12 +0100 you wrote:

Hi!

Even thought the __attribute__((patchable_function_entry())) solution to the BPF dispatcher woes works, it turns out to not be supported by the whole range of ageing compilers we support. Specifically this attribute seems to be GCC-8 and later.

[...]

Here is the summary with links:
- [1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") (no matching commit)
- [2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace)

    https://git.kernel.org/bpf/bpf/c/c86df29d11df

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html