Skip to contentSkip to conversation

bpf,ftrace: bpf dispatcher function fix

PATCH Applied Jiri Olsa 5 participants 7 messages Sep 3, 2022 – Sep 16, 2022

Subsystems (6)
lore
Share & tools
lore

Jiri Olsa

author reviewer
#0 lore raw

v3 cover bpf,ftrace: bpf dispatcher function fix

hi,
as discussed [1] sending fix that moves bpf dispatcher function of out ftrace locations together with Peter's HAVE_DYNAMIC_FTRACE_NO_PATCHABLE dependency change.

v3 changes:
- using notrace and adding it also for all archs [Peter]

v2 changes:
- fixing s390x CI build failure by enabling attributes only for x86_64 [Ilya Leoshkevic]

thanks,
jirka

[1] https://lore.kernel.org/bpf/[email protected]/
---
Jiri Olsa (1):
bpf: Move bpf_dispatcher function out of ftrace locations

Peter Zijlstra (Intel) (1):
ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE

 arch/x86/Kconfig                  |  1 +
 include/asm-generic/vmlinux.lds.h | 11 ++++++++++-
 include/linux/bpf.h               |  7 +++++++
 kernel/trace/Kconfig              |  6 ++++++
 tools/objtool/check.c             |  3 ++-
 5 files changed, 26 insertions(+), 2 deletions(-)
author reviewer
#1 lore raw

PATCH v3 1/2 ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE

From: "Peter Zijlstra (Intel)" <[email protected]>

x86 will shortly start using -fpatchable-function-entry for purposes other than ftrace, make sure the __patchable_function_entry section isn't merged in the mcount_loc section.

---
 include/asm-generic/vmlinux.lds.h | 11 ++++++++++-
 kernel/trace/Kconfig              |  6 ++++++
 tools/objtool/check.c             |  3 ++-
 3 files changed, 18 insertions(+), 2 deletions(-)
include/asm-generic/vmlinux.lds.h +10 -1
index 7515a465ec03..13b197ef0d63 100644
@@ -154,6 +154,14 @@
 #define MEM_DISCARD(sec) *(.mem##sec)
 #endif
 
+#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
+#define KEEP_PATCHABLE		KEEP(*(__patchable_function_entries))
+#define PATCHABLE_DISCARDS
+#else
+#define KEEP_PATCHABLE
+#define PATCHABLE_DISCARDS	*(__patchable_function_entries)
+#endif
+
 #ifdef CONFIG_FTRACE_MCOUNT_RECORD
 /*
  * The ftrace call sites are logged to a section whose name depends on the
@@ -172,7 +180,7 @@
 #define MCOUNT_REC()	. = ALIGN(8);				\
 			__start_mcount_loc = .;			\
 			KEEP(*(__mcount_loc))			\
-			KEEP(*(__patchable_function_entries))	\
+			KEEP_PATCHABLE				\
 			__stop_mcount_loc = .;			\
 			ftrace_stub_graph = ftrace_stub;	\
 			ftrace_ops_list_func = arch_ftrace_ops_list_func;
@@ -1024,6 +1032,7 @@
 
 #define COMMON_DISCARDS							\
 	SANITIZER_DISCARDS						\
+	PATCHABLE_DISCARDS						\
 	*(.discard)							\
 	*(.discard.*)							\
 	*(.modinfo)							\
kernel/trace/Kconfig +6 -0
index 1052126bdca2..e9e95c790b8e 100644
@@ -51,6 +51,12 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS
 	 This allows for use of regs_get_kernel_argument() and
 	 kernel_stack_pointer().
 
+config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
+	bool
+	help
+	  If the architecture generates __patchable_function_entries sections
+	  but does not want them included in the ftrace locations.
+
 config HAVE_FTRACE_MCOUNT_RECORD
 	bool
 	help
tools/objtool/check.c +2 -1
index 0cec74da7ffe..f23e8d11f6d4 100644
@@ -4110,7 +4110,8 @@ static int validate_ibt(struct objtool_file *file)
 		    !strcmp(sec->name, "__bug_table")			||
 		    !strcmp(sec->name, "__ex_table")			||
 		    !strcmp(sec->name, "__jump_table")			||
-		    !strcmp(sec->name, "__mcount_loc"))
+		    !strcmp(sec->name, "__mcount_loc")			||
+		    strstr(sec->name, "__patchable_function_entries"))
 			continue;
 
 		list_for_each_entry(reloc, &sec->reloc->reloc_list, list)
Signature · 1 line
2.37.2
author reviewer
#2 lore raw

PATCH v3 2/2 bpf: Move bpf_dispatcher function out of ftrace locations

In reply to Jiri Olsa

The dispatcher function is attached/detached to trampoline by dispatcher update function. At the same time it's available as ftrace attachable function.

After discussion [1] the proposed solution is to use compiler attributes to alter bpf_dispatcher_##name##_func function:

- remove it from being instrumented with __no_instrument_function__ attribute, so ftrace has no track of it

- but still generate 5 nop instructions with patchable_function_entry(5) attribute, which are expected by bpf_arch_text_poke used by dispatcher update function

Enabling HAVE_DYNAMIC_FTRACE_NO_PATCHABLE option for x86, so __patchable_function_entries functions are not part of ftrace/mcount locations.

Adding attributes to bpf_dispatcher_XXX function on x86_64 so it's kept out of ftrace locations and has 5 byte nop generated at entry.

These attributes need to be arch specific as pointer out by Ilya Leoshkevic in here [2].

The dispatcher image is generated only for x86_64 arch, so the code can stay as is for other archs.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

---
 arch/x86/Kconfig    | 1 +
 include/linux/bpf.h | 7 +++++++
 2 files changed, 8 insertions(+)
arch/x86/Kconfig +1 -0
index f9920f1341c8..089c20cefd2b 100644
@@ -284,6 +284,7 @@ config X86
 	select PROC_PID_ARCH_STATUS		if PROC_FS
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	imply IMA_SECURE_AND_OR_TRUSTED_BOOT    if EFI
+	select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
 
 config INSTRUCTION_DECODER
 	def_bool y
include/linux/bpf.h +7 -0
index 9c1674973e03..e267625557cb 100644
@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs);
 	},							\
 }
 
+#ifdef CONFIG_X86_64
+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
+#else
+#define BPF_DISPATCHER_ATTRIBUTES
+#endif
+
 #define DEFINE_BPF_DISPATCHER(name)					\
+	notrace BPF_DISPATCHER_ATTRIBUTES				\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
Signature · 1 line
2.37.2
maintainer
#3 lore raw

Re: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations

Quoted 17 lines · On Sat, 3 Sep 2022 15:11:54 +0200 Jiri Olsa <[email protected]> wrote:
arch/x86/Kconfig +1 -0
index f9920f1341c8..089c20cefd2b 100644
@@ -284,6 +284,7 @@ config X86
 	select PROC_PID_ARCH_STATUS		if PROC_FS
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	imply IMA_SECURE_AND_OR_TRUSTED_BOOT    if EFI
+	select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
 
 config INSTRUCTION_DECODER
 	def_bool y
include/linux/bpf.h +0 -0
index 9c1674973e03..e267625557cb 100644

include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 927

Quoted 4 lines
include/linux/bpf.h +1 -0
@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs);
 	},							\
 }
 
+#ifdef CONFIG_X86_64

I think Peter may have already mentioned this, but shouldn't he above be:

  #ifdef HAVE_DYNAMIC_FTRACE_NO_PATCHABLE

??

-- Steve

Quoted 11 lines
include/linux/bpf.h +6 -0
+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
+#else
+#define BPF_DISPATCHER_ATTRIBUTES
+#endif
+
 #define DEFINE_BPF_DISPATCHER(name)					\
+	notrace BPF_DISPATCHER_ATTRIBUTES				\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
#4 lore raw

Re: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations

Quoted 23 lines · Steven Rostedt
Quoted 20 lines · On Sat, 3 Sep 2022 15:11:54 +0200 Jiri Olsa <[email protected]> wrote:
arch/x86/Kconfig +1 -0
index f9920f1341c8..089c20cefd2b 100644
@@ -284,6 +284,7 @@ config X86
 	select PROC_PID_ARCH_STATUS		if PROC_FS
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	imply IMA_SECURE_AND_OR_TRUSTED_BOOT    if EFI
+	select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
 
 config INSTRUCTION_DECODER
 	def_bool y
include/linux/bpf.h +0 -0
index 9c1674973e03..e267625557cb 100644
@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs);
 	},							\
 }
 
Quoted 5 lines

include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 927

Quoted 1 line
include/linux/bpf.h +1 -0
+#ifdef CONFIG_X86_64

I think Peter may have already mentioned this, but shouldn't he above be:

  #ifdef HAVE_DYNAMIC_FTRACE_NO_PATCHABLE

different archs need different patchable_function_entry(X) attribute so I think we should use arch configs in here

also having HAVE_DYNAMIC_FTRACE_NO_PATCHABLE option enabled does not imply there's support for dispatcher image generation, so we might endup with extra nop bytes

jirka

Quoted 16 lines

??

-- Steve

Quoted 11 lines
include/linux/bpf.h +6 -0
+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
+#else
+#define BPF_DISPATCHER_ATTRIBUTES
+#endif
+
 #define DEFINE_BPF_DISPATCHER(name)					\
+	notrace BPF_DISPATCHER_ATTRIBUTES				\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
maintainer Acked-by on v3 2/2
#5 lore raw

Re: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations

In reply to Jiri Olsa
Quoted 57 lines · Jiri Olsa

The dispatcher function is attached/detached to trampoline by dispatcher update function. At the same time it's available as ftrace attachable function.

After discussion [1] the proposed solution is to use compiler attributes to alter bpf_dispatcher_##name##_func function:

- remove it from being instrumented with __no_instrument_function__ attribute, so ftrace has no track of it

- but still generate 5 nop instructions with patchable_function_entry(5) attribute, which are expected by bpf_arch_text_poke used by dispatcher update function

Enabling HAVE_DYNAMIC_FTRACE_NO_PATCHABLE option for x86, so __patchable_function_entries functions are not part of ftrace/mcount locations.

Adding attributes to bpf_dispatcher_XXX function on x86_64 so it's kept out of ftrace locations and has 5 byte nop generated at entry.

These attributes need to be arch specific as pointer out by Ilya Leoshkevic in here [2].

The dispatcher image is generated only for x86_64 arch, so the code can stay as is for other archs.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/

---
 arch/x86/Kconfig    | 1 +
 include/linux/bpf.h | 7 +++++++
 2 files changed, 8 insertions(+)
arch/x86/Kconfig +1 -0
index f9920f1341c8..089c20cefd2b 100644
@@ -284,6 +284,7 @@ config X86
 	select PROC_PID_ARCH_STATUS		if PROC_FS
 	select HAVE_ARCH_NODE_DEV_GROUP		if X86_SGX
 	imply IMA_SECURE_AND_OR_TRUSTED_BOOT    if EFI
+	select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
 
 config INSTRUCTION_DECODER
 	def_bool y
include/linux/bpf.h +1 -0
index 9c1674973e03..e267625557cb 100644
@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs);
 	},							\
 }
 
+#ifdef CONFIG_X86_64

include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 937

Quoted 10 lines
include/linux/bpf.h +6 -0
+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5)))
+#else
+#define BPF_DISPATCHER_ATTRIBUTES
+#endif
+
 #define DEFINE_BPF_DISPATCHER(name)					\
+	notrace BPF_DISPATCHER_ATTRIBUTES				\
 	noinline __nocfi unsigned int bpf_dispatcher_##name##_func(	\
 		const void *ctx,					\
 		const struct bpf_insn *insnsi,				\
#6 lore raw
In reply to Jiri Olsa
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:

Quoted 9 lines · On Sat, 3 Sep 2022 15:11:52 +0200 you wrote:

hi,
as discussed [1] sending fix that moves bpf dispatcher function of out ftrace locations together with Peter's HAVE_DYNAMIC_FTRACE_NO_PATCHABLE dependency change.

v3 changes:
- using notrace and adding it also for all archs [Peter]

[...]

Here is the summary with links:
- [PATCHv3,bpf-next,1/2] ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE

    https://git.kernel.org/bpf/bpf-next/c/9440155ccb94
  - [PATCHv3,bpf-next,2/2] bpf: Move bpf_dispatcher function out of ftrace locations
    https://git.kernel.org/bpf/bpf-next/c/ceea991a019c

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html