s390/ftrace: implement hotpatching
cover s390/ftrace: implement hotpatching
Hello,
This series implements the stop_machine()-less ftrace code patching for s390. It's trickier than that of e.g. Intel, because the only on-the-fly code patching allowed by s390 is changing a mask of a conditional branch instruction. Patch 1 contains a new common code hook that we need for that, patch 2 contains the actual implementation as well as explanation of all the small details.
We would like to take this series through the s390 tree, and we need a review on the patch 1 for that. Could you please take a look?
Best regards,
Ilya
Ilya Leoshkevich (2):
ftrace: Introduce ftrace_need_init_nop()
s390/ftrace: implement hotpatching
arch/s390/include/asm/ftrace.h | 46 +------ arch/s390/include/asm/ftrace.lds.h | 21 +++ arch/s390/include/asm/module.h | 8 ++ arch/s390/kernel/ftrace.c | 207 ++++++++++++++++++++++++++--- arch/s390/kernel/ftrace.h | 24 ++++ arch/s390/kernel/module.c | 45 +++++++ arch/s390/kernel/vmlinux.lds.S | 2 + include/linux/ftrace.h | 16 +++ kernel/trace/ftrace.c | 4 +- 9 files changed, 314 insertions(+), 59 deletions(-) create mode 100644 arch/s390/include/asm/ftrace.lds.h create mode 100644 arch/s390/kernel/ftrace.h
Signature · 1 line
2.31.1
PATCH 1/2 ftrace: Introduce ftrace_need_init_nop()
Implementing live patching on s390 requires each function's prologue to contain a very special kind of nop, which gcc and clang don't generate. However, the current code assumes that if CC_USING_NOP_MCOUNT is defined, then whatever the compiler generates is good enough.
Move the CC_USING_NOP_MCOUNT check into the new ftrace_need_init_nop() macro, that the architectures can override.
An alternative solution is to disable using -mnop-mcount in the Makefile, however, this makes the build logic (even) more complicated and forces the arch-specific code to deal with the useless __fentry__ symbol.
--- include/linux/ftrace.h | 16 ++++++++++++++++ kernel/trace/ftrace.c | 4 ++-- 2 files changed, 18 insertions(+), 2 deletions(-)
include/linux/ftrace.h +16 -0
@@ -643,6 +643,22 @@ static inline int ftrace_disable_ftrace_graph_caller(void) { return 0; }extern int ftrace_make_nop (struct module *mod,struct dyn_ftrace *rec,unsigned long addr); +/** +* ftrace_need_init_nop - return whether nop call sites should be initialized +* +* Normally the compiler's -mnop-mcount generates suitable nops, so we don't +* need to call ftrace_init_nop() if the code is built with that flag. +* Architectures where this is not always the case may define their own +* condition. +* +* Return must be: +* 0 if ftrace_init_nop() should be called +* Nonzero if ftrace_init_nop() should not be called +*/ + +#ifndef ftrace_need_init_nop +#define ftrace_need_init_nop () (!__is_defined(CC_USING_NOP_MCOUNT)) +#endif /** * ftrace_init_nop - initialize a nop call site
kernel/trace/ftrace.c +2 -2
@@ -3100,6 +3100,7 @@ ops_references_rec(struct ftrace_ops *ops, struct dyn_ftrace *rec)static int ftrace_update_code (struct module *mod,struct ftrace_page *new_pgs) { +bool init_nop =ftrace_need_init_nop ();struct ftrace_page *pg;struct dyn_ftrace *p;u64 start, stop;
@@ -3138,8 +3139,7 @@ static int ftrace_update_code(struct module *mod, struct ftrace_page *new_pgs)* Do the initial record conversion from mcount jump * to the NOP instructions. */ -if (!__is_defined (CC_USING_NOP_MCOUNT ) && - !ftrace_nop_initialize (mod, p)) +if (init_nop && !ftrace_nop_initialize (mod, p))break ; update_cnt++;
Signature · 1 line
2.31.1
Re: [PATCH 1/2] ftrace: Introduce ftrace_need_init_nop()
Quoted 8 lines · On Wed, 28 Jul 2021 23:25:45 +0200 Ilya Leoshkevich <[email protected]> wrote:
Implementing live patching on s390 requires each function's prologue to contain a very special kind of nop, which gcc and clang don't generate. However, the current code assumes that if
CC_USING_NOP_MCOUNTis defined, then whatever the compiler generates is good enough.Move the
CC_USING_NOP_MCOUNTcheck into the newftrace_need_init_nop()macro, that the architectures can override.
Quoted 6 lines
An alternative solution is to disable using -mnop-mcount in the
Makefile, however, this makes the build logic (even) more complicated and forces the arch-specific code to deal with the useless__fentry__symbol.Signed-off-by Ilya Leoshkevich <[email protected]>
I'm fine with this patch, and also you taking it in via the s390 tree.
-- Steve
Quoted 54 lines
--- include/linux/ftrace.h | 16 ++++++++++++++++ kernel/trace/ftrace.c | 4 ++-- 2 files changed, 18 insertions(+), 2 deletions(-)include/linux/ftrace.h +16
-0@@ -643,6 +643,22 @@ static inline int ftrace_disable_ftrace_graph_caller(void) { return 0; }extern int ftrace_make_nop (struct module *mod,struct dyn_ftrace *rec,unsigned long addr); +/** +* ftrace_need_init_nop - return whether nop call sites should be initialized +* +* Normally the compiler's -mnop-mcount generates suitable nops, so we don't +* need to call ftrace_init_nop() if the code is built with that flag. +* Architectures where this is not always the case may define their own +* condition. +* +* Return must be: +* 0 if ftrace_init_nop() should be called +* Nonzero if ftrace_init_nop() should not be called +*/ + +#ifndef ftrace_need_init_nop +#define ftrace_need_init_nop () (!__is_defined(CC_USING_NOP_MCOUNT)) +#endif /** * ftrace_init_nop - initialize a nop call site kernel/trace/ftrace.c +2
-2@@ -3100,6 +3100,7 @@ ops_references_rec(struct ftrace_ops *ops, struct dyn_ftrace *rec)static int ftrace_update_code (struct module *mod,struct ftrace_page *new_pgs) { +bool init_nop =ftrace_need_init_nop ();struct ftrace_page *pg;struct dyn_ftrace *p;u64 start, stop;@@ -3138,8 +3139,7 @@ static int ftrace_update_code(struct module *mod, struct ftrace_page *new_pgs)* Do the initial record conversion from mcount jump * to the NOP instructions. */ -if (!__is_defined (CC_USING_NOP_MCOUNT ) && - !ftrace_nop_initialize (mod, p)) +if (init_nop && !ftrace_nop_initialize (mod, p))break ; update_cnt++;
Heiko Carstens
maintainerRe: [PATCH 1/2] ftrace: Introduce ftrace_need_init_nop()
Quoted 16 lines · Steven Rostedt
Quoted 13 lines · On Wed, 28 Jul 2021 23:25:45 +0200 Ilya Leoshkevich <[email protected]> wrote:
Implementing live patching on s390 requires each function's prologue to contain a very special kind of nop, which gcc and clang don't generate. However, the current code assumes that if
CC_USING_NOP_MCOUNTis defined, then whatever the compiler generates is good enough.Move the
CC_USING_NOP_MCOUNTcheck into the newftrace_need_init_nop()macro, that the architectures can override.An alternative solution is to disable using -mnop-mcount in the
Makefile, however, this makes the build logic (even) more complicated and forces the arch-specific code to deal with the useless__fentry__symbol.
Quoted 5 lines
Quoted 1 line
Signed-off-by Ilya Leoshkevich <[email protected]>I'm fine with this patch, and also you taking it in via the s390 tree.
Reviewed-by Steven Rostedt (VMware) <[email protected]>
Thanks a lot!
PATCH 2/2 s390/ftrace: implement hotpatching
In reply to Ilya Leoshkevichs390 allows hotpatching the mask of a conditional jump instruction. Make use of this feature in order to avoid the expensive stop_machine() call.
The new trampolines are split in 3 stages:
- A first stage is a 6-byte relative conditional long branch located at each function's entry point. Its offset always points to the second stage for the corresponding function, and its mask is either all 0s (ftrace off) or all 1s (ftrace on). The code for flipping the mask is borrowed from ftrace_{enable,disable}_ftrace_graph_caller. After flipping, ftrace_arch_code_modify_post_process() syncs with all the other CPUs by sending SIGPs.
- Second stages for vmlinux are stored in a separate part of the .text section reserved by the linker script, and in dynamically allocated memory for modules. This prevents the icache pollution. The total size of second stages is about 1.5% of that of the kernel image.
Putting second stages in the .bss section is possible and decreases the size of the non-compressed vmlinux, but splits the kernel 1:1 mapping, which is a bad tradeoff.
Each second stage contains a call to the third stage, a pointer to the part of the intercepted function right after the first stage, and a pointer to an interceptor function (e.g. ftrace_caller).
Second stages are 8-byte aligned for the future direct calls implementation.
- There are only two copies of the third stage: in the .text section for vmlinux and in dynamically allocated memory for modules. It can be an expoline, which is relatively large, so inlining it into each second stage is prohibitively expensive.
As a result of this organization, phoronix-test-suite with ftrace off does not show any performance degradation.
At least a couple things are missing right now, but can be implemented separately later:
- Expoline version of the third stage is supported only for z10+. Implementing it for earlier machines is possible, but is relatively complicated due to the lack of available registers.
- On z15+ trampolines may use a BIC instruction, which loads an address from memory and jumps to it.
--- arch/s390/include/asm/ftrace.h | 46 +------ arch/s390/include/asm/ftrace.lds.h | 21 +++ arch/s390/include/asm/module.h | 8 ++ arch/s390/kernel/ftrace.c | 207 ++++++++++++++++++++++++++--- arch/s390/kernel/ftrace.h | 24 ++++ arch/s390/kernel/module.c | 45 +++++++ arch/s390/kernel/vmlinux.lds.S | 2 + 7 files changed, 296 insertions(+), 57 deletions(-) create mode 100644 arch/s390/include/asm/ftrace.lds.h create mode 100644 arch/s390/kernel/ftrace.h
arch/s390/include/asm/ftrace.h +5 -41
@@ -18,7 +18,6 @@void ftrace_caller (void );extern char ftrace_graph_caller_end; -extern unsigned long ftrace_plt;extern void *ftrace_func;struct dyn_arch_ftrace { };
@@ -31,10 +30,11 @@ struct dyn_arch_ftrace { };struct module ;struct dyn_ftrace ; -/* -* Either -mhotpatch or -mnop-mcount is used - no explicit init is required -*/ -static inline int ftrace_init_nop (struct module *mod,struct dyn_ftrace *rec) {return 0 ; } + +bool ftrace_need_init_nop (void ); +#define ftrace_need_init_nop ftrace_need_init_nop + +int ftrace_init_nop (struct module *mod,struct dyn_ftrace *rec);#define ftrace_init_nop ftrace_init_nopstatic inline unsigned long ftrace_call_adjust (unsigned long addr)
@@ -42,42 +42,6 @@ static inline unsigned long ftrace_call_adjust(unsigned long addr)return addr; } -struct ftrace_insn { -u16 opc; -s32 disp; -} __packed; - -static inline void ftrace_generate_nop_insn (struct ftrace_insn *insn) -{ -#ifdef CONFIG_FUNCTION_TRACER -/* brcl 0,0 */ - insn->opc =0xc004 ; - insn->disp =0 ; -#endif -} - -static inline int is_ftrace_nop (struct ftrace_insn *insn) -{ -#ifdef CONFIG_FUNCTION_TRACER -if (insn->disp ==0 ) -return 1 ; -#endif -return 0 ; -} - -static inline void ftrace_generate_call_insn (struct ftrace_insn *insn, -unsigned long ip) -{ -#ifdef CONFIG_FUNCTION_TRACER -unsigned long target; - -/* brasl r0,ftrace_caller */ - target =is_module_addr ((void *) ip) ? ftrace_plt :FTRACE_ADDR ; - insn->opc =0xc005 ; - insn->disp = (target - ip) /2 ; -#endif -} -/* * Even though the system call numbers are identical for s390/s390x a * different system call table is used for compat tasks. This may lead
arch/s390/include/asm/ftrace.lds.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef DIV_ROUND_UP +#define DIV_ROUND_UP (n, d) (((n) + (d) - 1) / (d)) +#endif + +#define SIZEOF_MCOUNT_LOC_ENTRY 8 +#define SIZEOF_FTRACE_HOTPATCH_TRAMPOLINE 24 +#define FTRACE_HOTPATCH_TRAMPOLINES_SIZE (n) \ + DIV_ROUND_UP(SIZEOF_FTRACE_HOTPATCH_TRAMPOLINE * (n), \ + SIZEOF_MCOUNT_LOC_ENTRY) + +#ifdef CONFIG_FUNCTION_TRACER +#define FTRACE_HOTPATCH_TRAMPOLINES_TEXT \ + . = ALIGN(8); \ + __ftrace_hotpatch_trampolines_start = .; \ + . = . + FTRACE_HOTPATCH_TRAMPOLINES_SIZE(__stop_mcount_loc - \ + __start_mcount_loc); \ + __ftrace_hotpatch_trampolines_end = .; +#else +#define FTRACE_HOTPATCH_TRAMPOLINES_TEXT +#endif
arch/s390/include/asm/module.h +8 -0
@@ -28,6 +28,14 @@ struct mod_arch_specific {int nsyms;/* Additional symbol information (got and plt offsets). */ struct mod_arch_syminfo *syminfo; +#ifdef CONFIG_FUNCTION_TRACER +/* Start of memory reserved for ftrace hotpatch trampolines. */ +struct ftrace_hotpatch_trampoline *trampolines_start; +/* End of memory reserved for ftrace hotpatch trampolines. */ +struct ftrace_hotpatch_trampoline *trampolines_end; +/* Next unused ftrace hotpatch trampoline slot. */ +struct ftrace_hotpatch_trampoline *next_trampoline; +#endif /* CONFIG_FUNCTION_TRACER */ };#endif /* _ASM_S390_MODULE_H */
arch/s390/kernel/ftrace.c +191 -16
266 lines not on this page · raw message
arch/s390/kernel/ftrace.h new file +24 -0
@@ -0,0 +1,24 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _FTRACE_H +#define _FTRACE_H + +#include <asm/types.h> + +struct ftrace_hotpatch_trampoline { +u16 brasl_opc; +s32 brasl_disp; + s16: 16; +u64 rest_of_intercepted_function; +u64 interceptor; +} __packed; + +extern struct ftrace_hotpatch_trampoline __ftrace_hotpatch_trampolines_start[]; +extern struct ftrace_hotpatch_trampoline __ftrace_hotpatch_trampolines_end[]; +extern const char ftrace_shared_hotpatch_trampoline_br[]; +extern const char ftrace_shared_hotpatch_trampoline_br_end[]; +extern const char ftrace_shared_hotpatch_trampoline_exrl[]; +extern const char ftrace_shared_hotpatch_trampoline_exrl_end[]; +extern const char ftrace_plt_template[]; +extern const char ftrace_plt_template_end[]; + +#endif /* _FTRACE_H */
arch/s390/kernel/module.c +45 -0
81 lines not on this page · raw message
arch/s390/kernel/vmlinux.lds.S +2 -0
@@ -5,6 +5,7 @@ #include <asm/thread_info.h> #include <asm/page.h> +#include <asm/ftrace.lds.h> /* * Put .bss..swapper_pg_dir as the first thing in .bss. This will
@@ -46,6 +47,7 @@ SECTIONS KPROBES_TEXT IRQENTRY_TEXT SOFTIRQENTRY_TEXT + FTRACE_HOTPATCH_TRAMPOLINES_TEXT *(.text.*_indirect_*) *(.fixup) *(.gnu.warning)
Signature · 1 line
2.31.1
Heiko Carstens
maintainerQuoted 19 lines · Ilya Leoshkevich
Hello,
This series implements the stop_machine()-less ftrace code patching for s390. It's trickier than that of e.g. Intel, because the only on-the-fly code patching allowed by s390 is changing a mask of a conditional branch instruction. Patch 1 contains a new common code hook that we need for that, patch 2 contains the actual implementation as well as explanation of all the small details.
We would like to take this series through the s390 tree, and we need a review on the patch 1 for that. Could you please take a look?
Best regards,
IlyaIlya Leoshkevich (2):
ftrace: Introduceftrace_need_init_nop()
s390/ftrace: implement hotpatching
Quoted 12 lines
arch/s390/include/asm/ftrace.h | 46 +------ arch/s390/include/asm/ftrace.lds.h | 21 +++ arch/s390/include/asm/module.h | 8 ++ arch/s390/kernel/ftrace.c | 207 ++++++++++++++++++++++++++--- arch/s390/kernel/ftrace.h | 24 ++++ arch/s390/kernel/module.c | 45 +++++++ arch/s390/kernel/vmlinux.lds.S | 2 + include/linux/ftrace.h | 16 +++ kernel/trace/ftrace.c | 4 +- 9 files changed, 314 insertions(+), 59 deletions(-) create mode 100644 arch/s390/include/asm/ftrace.lds.h create mode 100644 arch/s390/kernel/ftrace.h
Series applied, thank you!