bpf,ftrace: bpf dispatcher function fix
Subsystems (6)
Jiri Olsa
reviewerv3 cover bpf,ftrace: bpf dispatcher function fix
hi,
as discussed [1] sending fix that moves bpf dispatcher function of out ftrace locations together with Peter's HAVE_DYNAMIC_FTRACE_NO_PATCHABLE dependency change.
v3 changes:
- using notrace and adding it also for all archs [Peter]
v2 changes:
- fixing s390x CI build failure by enabling attributes only for x86_64 [Ilya Leoshkevic]
thanks,
jirka
[1] https://lore.kernel.org/bpf/[email protected]/
---
Jiri Olsa (1):
bpf: Move bpf_dispatcher function out of ftrace locations
Peter Zijlstra (Intel) (1):
ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
arch/x86/Kconfig | 1 + include/asm-generic/vmlinux.lds.h | 11 ++++++++++- include/linux/bpf.h | 7 +++++++ kernel/trace/Kconfig | 6 ++++++ tools/objtool/check.c | 3 ++- 5 files changed, 26 insertions(+), 2 deletions(-)
Jiri Olsa
reviewerPATCH v3 1/2 ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
From: "Peter Zijlstra (Intel)" <[email protected]>
x86 will shortly start using -fpatchable-function-entry for purposes other than ftrace, make sure the __patchable_function_entry section isn't merged in the mcount_loc section.
--- include/asm-generic/vmlinux.lds.h | 11 ++++++++++- kernel/trace/Kconfig | 6 ++++++ tools/objtool/check.c | 3 ++- 3 files changed, 18 insertions(+), 2 deletions(-)
include/asm-generic/vmlinux.lds.h +10 -1
@@ -154,6 +154,14 @@#define MEM_DISCARD (sec) *(.mem##sec)#endif +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_NO_PATCHABLE +#define KEEP_PATCHABLE KEEP(*(__patchable_function_entries)) +#define PATCHABLE_DISCARDS +#else +#define KEEP_PATCHABLE +#define PATCHABLE_DISCARDS *(__patchable_function_entries) +#endif +#ifdef CONFIG_FTRACE_MCOUNT_RECORD /* * The ftrace call sites are logged to a section whose name depends on the
@@ -172,7 +180,7 @@#define MCOUNT_REC () . = ALIGN(8); \ __start_mcount_loc = .; \ KEEP(*(__mcount_loc)) \ - KEEP(*(__patchable_function_entries)) \ + KEEP_PATCHABLE \ __stop_mcount_loc = .; \ ftrace_stub_graph = ftrace_stub; \ ftrace_ops_list_func = arch_ftrace_ops_list_func;
@@ -1024,6 +1032,7 @@#define COMMON_DISCARDS \ SANITIZER_DISCARDS \ + PATCHABLE_DISCARDS \ *(.discard) \ *(.discard.*) \ *(.modinfo) \
kernel/trace/Kconfig +6 -0
@@ -51,6 +51,12 @@ config HAVE_DYNAMIC_FTRACE_WITH_ARGS This allows for use of regs_get_kernel_argument() and kernel_stack_pointer(). +config HAVE_DYNAMIC_FTRACE_NO_PATCHABLE + bool + help + If the architecture generates __patchable_function_entries sections + but does not want them included in the ftrace locations. + config HAVE_FTRACE_MCOUNT_RECORD bool help
tools/objtool/check.c +2 -1
@@ -4110,7 +4110,8 @@ static int validate_ibt(struct objtool_file *file) !strcmp (sec->name,"__bug_table" ) || !strcmp (sec->name,"__ex_table" ) || !strcmp (sec->name,"__jump_table" ) || - !strcmp(sec->name, "__mcount_loc")) + !strcmp (sec->name,"__mcount_loc" ) || + strstr(sec->name, "__patchable_function_entries")) continue;list_for_each_entry (reloc, &sec->reloc->reloc_list, list)
Signature · 1 line
2.37.2
Jiri Olsa
reviewerPATCH v3 2/2 bpf: Move bpf_dispatcher function out of ftrace locations
In reply to Jiri OlsaThe dispatcher function is attached/detached to trampoline by dispatcher update function. At the same time it's available as ftrace attachable function.
After discussion [1] the proposed solution is to use compiler attributes to alter bpf_dispatcher_##name##_func function:
- remove it from being instrumented with __no_instrument_function__ attribute, so ftrace has no track of it
- but still generate 5 nop instructions with patchable_function_entry(5) attribute, which are expected by bpf_arch_text_poke used by dispatcher update function
Enabling HAVE_DYNAMIC_FTRACE_NO_PATCHABLE option for x86, so __patchable_function_entries functions are not part of ftrace/mcount locations.
Adding attributes to bpf_dispatcher_XXX function on x86_64 so it's kept out of ftrace locations and has 5 byte nop generated at entry.
These attributes need to be arch specific as pointer out by Ilya Leoshkevic in here [2].
The dispatcher image is generated only for x86_64 arch, so the code can stay as is for other archs.
[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
--- arch/x86/Kconfig | 1 + include/linux/bpf.h | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/Kconfig +1 -0
@@ -284,6 +284,7 @@ config X86 select PROC_PID_ARCH_STATUS if PROC_FS select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE config INSTRUCTION_DECODER def_bool y
include/linux/bpf.h +7 -0
@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); }, \ } +#ifdef CONFIG_X86_64 +#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) +#else +#define BPF_DISPATCHER_ATTRIBUTES +#endif +#define DEFINE_BPF_DISPATCHER (name) \ + notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \
Signature · 1 line
2.37.2
Steven Rostedt
maintainerRe: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations
Quoted 17 lines · On Sat, 3 Sep 2022 15:11:54 +0200 Jiri Olsa <[email protected]> wrote:
arch/x86/Kconfig +1
-0@@ -284,6 +284,7 @@ config X86 select PROC_PID_ARCH_STATUS if PROC_FS select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE config INSTRUCTION_DECODER def_bool yinclude/linux/bpf.h +0
-0
include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 927
Quoted 4 lines
include/linux/bpf.h +1
-0@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); }, \ } +#ifdef CONFIG_X86_64
I think Peter may have already mentioned this, but shouldn't he above be:
#ifdef HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
??
-- Steve
Quoted 11 lines
include/linux/bpf.h +6
-0+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) +#else +#define BPF_DISPATCHER_ATTRIBUTES +#endif +#define DEFINE_BPF_DISPATCHER (name) \ + notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \
Re: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations
Quoted 23 lines · Steven Rostedt
Quoted 20 lines · On Sat, 3 Sep 2022 15:11:54 +0200 Jiri Olsa <[email protected]> wrote:
arch/x86/Kconfig +1
-0@@ -284,6 +284,7 @@ config X86 select PROC_PID_ARCH_STATUS if PROC_FS select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE config INSTRUCTION_DECODER def_bool yinclude/linux/bpf.h +0
-0@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); }, \ }
Quoted 5 lines
include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 927
Quoted 1 line
include/linux/bpf.h +1
-0+#ifdef CONFIG_X86_64 I think Peter may have already mentioned this, but shouldn't he above be:
#ifdef HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
different archs need different patchable_function_entry(X) attribute so I think we should use arch configs in here
also having HAVE_DYNAMIC_FTRACE_NO_PATCHABLE option enabled does not imply there's support for dispatcher image generation, so we might endup with extra nop bytes
jirka
Quoted 16 lines
??
-- Steve
Quoted 11 lines
include/linux/bpf.h +6
-0+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) +#else +#define BPF_DISPATCHER_ATTRIBUTES +#endif +#define DEFINE_BPF_DISPATCHER (name) \ + notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \
Re: [PATCHv3 bpf-next 2/2] bpf: Move bpf_dispatcher function out of ftrace locations
In reply to Jiri OlsaQuoted 57 lines · Jiri Olsa
The dispatcher function is attached/detached to trampoline by dispatcher update function. At the same time it's available as ftrace attachable function.
After discussion [1] the proposed solution is to use compiler attributes to alter bpf_dispatcher_##name##_func function:
- remove it from being instrumented with
__no_instrument_function__attribute, so ftrace has no track of it- but still generate 5 nop instructions with
patchable_function_entry(5)attribute, which are expected bybpf_arch_text_pokeused by dispatcher update functionEnabling
HAVE_DYNAMIC_FTRACE_NO_PATCHABLEoption for x86, so__patchable_function_entriesfunctions are not part of ftrace/mcount locations.Adding attributes to
bpf_dispatcher_XXXfunction onx86_64so it's kept out of ftrace locations and has 5 byte nop generated at entry.These attributes need to be arch specific as pointer out by Ilya Leoshkevic in here [2].
The dispatcher image is generated only for
x86_64arch, so the code can stay as is for other archs.[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/Suggested-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Jiri Olsa <[email protected]>--- arch/x86/Kconfig | 1 + include/linux/bpf.h | 7 +++++++ 2 files changed, 8 insertions(+)arch/x86/Kconfig +1
-0@@ -284,6 +284,7 @@ config X86 select PROC_PID_ARCH_STATUS if PROC_FS select HAVE_ARCH_NODE_DEV_GROUP if X86_SGX imply IMA_SECURE_AND_OR_TRUSTED_BOOT if EFI + select HAVE_DYNAMIC_FTRACE_NO_PATCHABLE config INSTRUCTION_DECODER def_bool yinclude/linux/bpf.h +1
-0@@ -924,7 +924,14 @@ int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); }, \ } +#ifdef CONFIG_X86_64
include/linux/bpf.h int arch_prepare_bpf_dispatcher(void *image, s64 *funcs, int num_funcs); · line 937
Quoted 10 lines
include/linux/bpf.h +6
-0+#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) +#else +#define BPF_DISPATCHER_ATTRIBUTES +#endif +#define DEFINE_BPF_DISPATCHER (name) \ + notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Daniel Borkmann <[email protected]>:
Quoted 9 lines · On Sat, 3 Sep 2022 15:11:52 +0200 you wrote:
hi,
as discussed [1] sending fix that moves bpf dispatcher function of out ftrace locations together with Peter'sHAVE_DYNAMIC_FTRACE_NO_PATCHABLEdependency change.v3 changes:
- using notrace and adding it also for all archs [Peter][...]
Here is the summary with links:
- [PATCHv3,bpf-next,1/2] ftrace: Add HAVE_DYNAMIC_FTRACE_NO_PATCHABLE
https://git.kernel.org/bpf/bpf-next/c/9440155ccb94 - [PATCHv3,bpf-next,2/2] bpf: Move bpf_dispatcher function out of ftrace locations https://git.kernel.org/bpf/bpf-next/c/ceea991a019c
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html