x86/ftrace: Add direct batch interface
v2 cover x86/ftrace: Add direct batch interface
hi,
adding interface to maintain multiple direct functions within single calls. It's a base for follow up bpf batch attach functionality.
New interface:
int register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) int unregister_ftrace_direct_multi(struct ftrace_ops *ops)
int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)
that allows to register/unregister/modify direct function 'addr' with struct ftrace_ops object. The ops filter can be updated before with ftrace_set_filter_ip calls
1) patches (1-4) that fix the ftrace graph tracing over the function with direct trampolines attached
2) patches (5-8) that add batch interface for ftrace direct function register/unregister/modify
Also available at (based on Steven's ftrace/core branch):
https://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
ftrace/direct
v2 changes:
- added comments to API functions
- move common cleanup code to new remove_direct_functions_hash function
- added addr argument to unregister_ftrace_direct_multi API, to properly cleanup direct_functions hash
- added missing unregister_ftrace_direct call to trampoline/graph selftest
- added comment to modify_ftrace_direct_multi about non-callback gap between ftrace_shutdown and ftrace_startup
thanks,
jirka
---
Jiri Olsa (6):
x86/ftrace: Remove extra orig rax move
tracing: Add trampoline/graph selftest
ftrace: Add ftrace_add_rec_direct function
ftrace: Add multi direct register/unregister interface
ftrace: Add multi direct modify interface ftrace/samples: Add multi direct interface test module
Steven Rostedt (VMware) (2):
x86/ftrace: Remove fault protection code in prepare_ftrace_return x86/ftrace: Make function graph use ftrace directly
arch/x86/include/asm/ftrace.h | 9 +++- arch/x86/kernel/ftrace.c | 71 +++++++++++++++--------------- arch/x86/kernel/ftrace_64.S | 30 +------------ include/linux/ftrace.h | 26 +++++++++++ kernel/trace/fgraph.c | 6 ++- kernel/trace/ftrace.c | 268 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------ kernel/trace/trace_selftest.c | 54 ++++++++++++++++++++++- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++ 9 files changed, 420 insertions(+), 97 deletions(-) create mode 100644 samples/ftrace/ftrace-direct-multi.c
PATCH v2 1/8 x86/ftrace: Remove extra orig rax move
There's identical move 2 lines earlier.
--- arch/x86/kernel/ftrace_64.S | 1 - 1 file changed, 1 deletion(-)
arch/x86/kernel/ftrace_64.S +0 -1
@@ -251,7 +251,6 @@ SYM_INNER_LABEL(ftrace_regs_call, SYM_L_GLOBAL) * If ORIG_RAX is anything but zero, make this a call to that. * See arch_ftrace_set_direct_caller(). */ - movq ORIG_RAX(%rsp), %rax testq %rax, %rax SYM_INNER_LABEL(ftrace_regs_caller_jmp, SYM_L_GLOBAL) jnz 1f
Signature · 1 line
2.31.1
PATCH v2 2/8 x86/ftrace: Remove fault protection code in prepare_ftrace_return
In reply to Jiri OlsaFrom: "Steven Rostedt (VMware)" <[email protected]>
Removing the fault protection code when writing return_hooker to stack. As Steven noted:
Quoted 3 lines
That protection was there from the beginning due to being "paranoid", considering ftrace was bricking network cards. But that protection would not have even protected against that.
--- arch/x86/kernel/ftrace.c | 38 +++----------------------------------- 1 file changed, 3 insertions(+), 35 deletions(-)
arch/x86/kernel/ftrace.c +3 -35
@@ -625,12 +625,10 @@ int ftrace_disable_ftrace_graph_caller(void)* Hook the return address and push it in the stack of return addrs * in current thread info. */ -void prepare_ftrace_return (unsigned long self_addr,unsigned long *parent, +void prepare_ftrace_return (unsigned long ip,unsigned long *parent,unsigned long frame_pointer) {unsigned long return_hooker = (unsigned long )&return_to_handler; -unsigned long old; -int faulted;/* * When resuming from suspend-to-ram, this function can be indirectly
@@ -650,37 +648,7 @@ void prepare_ftrace_return(unsigned long self_addr, unsigned long *parent,if (unlikely (atomic_read (¤t->tracing_graph_pause)))return ; -/* -* Protect against fault, even if it shouldn't -* happen. This tool is too much intrusive to -* ignore such a protection. -*/ - asmvolatile ( -"1: " _ASM_MOV" (%[parent]), %[old]\n" -"2: " _ASM_MOV" %[return_hooker], (%[parent])\n" -" movl $0, %[faulted]\n" -"3:\n" - -".section .fixup, \"ax\"\n" -"4: movl $1, %[faulted]\n" -" jmp 3b\n" -".previous\n" - - _ASM_EXTABLE(1b, 4b) -_ASM_EXTABLE (2b ,4b ) - - : [old] "=&r" (old), [faulted] "=r" (faulted) - : [parent] "r" (parent), [return_hooker] "r" (return_hooker) - : "memory" - ); - -if (unlikely (faulted)) { -ftrace_graph_stop (); -WARN_ON (1 ); -return ; - } - -if (function_graph_enter (old, self_addr, frame_pointer, parent)) - *parent = old; +if (!function_graph_enter (*parent, ip, frame_pointer, parent)) + *parent = return_hooker; }#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
Signature · 1 line
2.31.1
PATCH v2 3/8 x86/ftrace: Make function graph use ftrace directly
In reply to Jiri OlsaFrom: "Steven Rostedt (VMware)" <[email protected]>
We don't need special hook for graph tracer entry point, but instead we can use graph_ops::func function to install the return_hooker.
This moves the graph tracing setup _before_ the direct trampoline prepares the stack, so the return_hooker will be called when the direct trampoline is finished.
This simplifies the code, because we don't need to take into account the direct trampoline setup when preparing the graph tracer hooker and we can allow function graph tracer on entries registered with direct trampoline.
--- arch/x86/include/asm/ftrace.h | 9 +++++++-- arch/x86/kernel/ftrace.c | 37 ++++++++++++++++++++++++++++++++--- arch/x86/kernel/ftrace_64.S | 29 +-------------------------- include/linux/ftrace.h | 9 +++++++++ kernel/trace/fgraph.c | 6 ++++-- 5 files changed, 55 insertions(+), 35 deletions(-)
arch/x86/include/asm/ftrace.h +7 -2
@@ -57,6 +57,13 @@ arch_ftrace_get_regs(struct ftrace_regs *fregs)#define ftrace_instruction_pointer_set (fregs, _ip) \ do { (fregs)->regs.ip = (_ip); } while (0) + +struct ftrace_ops ; +#define ftrace_graph_func ftrace_graph_func +void ftrace_graph_func (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *op,struct ftrace_regs *fregs); +#else +#define FTRACE_GRAPH_TRAMP_ADDR FTRACE_GRAPH_ADDR#endif #ifdef CONFIG_DYNAMIC_FTRACE
@@ -65,8 +72,6 @@ struct dyn_arch_ftrace {/* No extra data needed for x86 */ }; -#define FTRACE_GRAPH_TRAMP_ADDR FTRACE_GRAPH_ADDR -#endif /* CONFIG_DYNAMIC_FTRACE */ #endif /* __ASSEMBLY__ */ #endif /* CONFIG_FUNCTION_TRACER */
arch/x86/kernel/ftrace.c +34 -3
@@ -527,7 +527,7 @@ static void *addr_from_call(void *ptr)return ptr +CALL_INSN_SIZE + call.disp; } -void prepare_ftrace_return (unsigned long self_addr,unsigned long *parent, +void prepare_ftrace_return (unsigned long ip,unsigned long *parent,unsigned long frame_pointer);/*
@@ -541,7 +541,8 @@ static void *static_tramp_func(struct ftrace_ops *ops, struct dyn_ftrace *rec)void *ptr;if (ops && ops->trampoline) { -#ifdef CONFIG_FUNCTION_GRAPH_TRACER +#if !defined(CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS ) && \ + defined(CONFIG_FUNCTION_GRAPH_TRACER )/* * We only know about function graph tracer setting as static * trampoline.
@@ -589,8 +590,9 @@ void arch_ftrace_trampoline_free(struct ftrace_ops *ops) #ifdef CONFIG_FUNCTION_GRAPH_TRACER #ifdef CONFIG_DYNAMIC_FTRACE -extern void ftrace_graph_call(void); +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +extern void ftrace_graph_call(void); static const char *ftrace_jmp_replace(unsigned long ip, unsigned long addr) { return text_gen_insn(JMP32_INSN_OPCODE, (void *)ip, (void *)addr);
@@ -618,7 +620,17 @@ int ftrace_disable_ftrace_graph_caller(void)return ftrace_mod_jmp (ip, &ftrace_stub); } +#else /* !CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ +int ftrace_enable_ftrace_graph_caller (void ) +{ +return 0 ; +} +int ftrace_disable_ftrace_graph_caller (void ) +{ +return 0 ; +} +#endif /* CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS */ #endif /* !CONFIG_DYNAMIC_FTRACE */ /*
@@ -629,6 +641,7 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent, unsigned long frame_pointer) {unsigned long return_hooker = (unsigned long )&return_to_handler; +int bit;/* * When resuming from suspend-to-ram, this function can be indirectly
@@ -648,7 +661,25 @@ void prepare_ftrace_return(unsigned long ip, unsigned long *parent,if (unlikely (atomic_read (¤t->tracing_graph_pause)))return ; + bit =ftrace_test_recursion_trylock (ip, *parent); +if (bit <0 ) +return ; +if (!function_graph_enter (*parent, ip, frame_pointer, parent)) *parent = return_hooker; + + ftrace_test_recursion_unlock(bit); +} + +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +void ftrace_graph_func (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *op,struct ftrace_regs *fregs) +{ +struct pt_regs *regs = &fregs->regs; +unsigned long *stack = (unsigned long *)kernel_stack_pointer (regs); + +prepare_ftrace_return (ip, (unsigned long *)stack,0 ); } +#endif +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
arch/x86/kernel/ftrace_64.S +1 -28
@@ -174,11 +174,6 @@ SYM_INNER_LABEL(ftrace_caller_end, SYM_L_GLOBAL) SYM_FUNC_END(ftrace_caller); SYM_FUNC_START(ftrace_epilogue) -#ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_INNER_LABEL(ftrace_graph_call, SYM_L_GLOBAL) - jmp ftrace_stub -#endif - /* * This is weak to keep gas from relaxing the jumps. * It is also used to copy the retq for trampolines.
@@ -288,15 +283,6 @@ SYM_FUNC_START(__fentry__) cmpq $ftrace_stub, ftrace_trace_function jnz trace -fgraph_trace: -#ifdef CONFIG_FUNCTION_GRAPH_TRACER - cmpq $ftrace_stub, ftrace_graph_return - jnz ftrace_graph_caller - - cmpq $ftrace_graph_entry_stub, ftrace_graph_entry - jnz ftrace_graph_caller -#endif - SYM_INNER_LABEL(ftrace_stub, SYM_L_GLOBAL) retq
@@ -314,25 +300,12 @@ trace: CALL_NOSPEC r8 restore_mcount_regs - jmp fgraph_trace + jmp ftrace_stub SYM_FUNC_END(__fentry__) EXPORT_SYMBOL(__fentry__) #endif /* CONFIG_DYNAMIC_FTRACE */ #ifdef CONFIG_FUNCTION_GRAPH_TRACER -SYM_FUNC_START(ftrace_graph_caller) - /* Saves rbp into %rdx and fills first parameter */ - save_mcount_regs - - leaq MCOUNT_REG_SIZE+8(%rsp), %rsi - movq $0, %rdx /* No framepointers needed */ - call prepare_ftrace_return - - restore_mcount_regs - - retq -SYM_FUNC_END(ftrace_graph_caller) - SYM_FUNC_START(return_to_handler) subq $24, %rsp
include/linux/ftrace.h +9 -0
@@ -795,6 +795,15 @@ static inline bool is_ftrace_trampoline(unsigned long addr) }#endif /* CONFIG_DYNAMIC_FTRACE */ +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifndef ftrace_graph_func +#define ftrace_graph_func ftrace_stub +#define FTRACE_OPS_GRAPH_STUB FTRACE_OPS_FL_STUB +#else +#define FTRACE_OPS_GRAPH_STUB 0 +#endif +#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ +/* totally disable ftrace - can not re-enable after this */ void ftrace_kill (void );
kernel/trace/fgraph.c +4 -2
@@ -115,6 +115,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, {struct ftrace_graph_ent trace; +#ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS /* * Skip graph tracing if the return location is served by direct trampoline, * since call sequence and return addresses are unpredictable anyway.
@@ -124,6 +125,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (ftrace_direct_func_count &&ftrace_find_rec_direct (ret -MCOUNT_INSN_SIZE ))return -EBUSY ; +#endif trace.func = func; trace.depth = ++current->curr_ret_depth;
@@ -333,10 +335,10 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx,#endif /* HAVE_FUNCTION_GRAPH_RET_ADDR_PTR */ static struct ftrace_ops graph_ops = { - .func = ftrace_stub, + .func = ftrace_graph_func, .flags =FTRACE_OPS_FL_INITIALIZED |FTRACE_OPS_FL_PID | -FTRACE_OPS_FL_STUB , +FTRACE_OPS_GRAPH_STUB , #ifdef FTRACE_GRAPH_TRAMP_ADDR .trampoline =FTRACE_GRAPH_TRAMP_ADDR , /* trampoline_size is only needed for dynamically allocated tramps */
Signature · 1 line
2.31.1
PATCH v2 4/8 tracing: Add trampoline/graph selftest
In reply to Jiri OlsaAdding selftest for checking that direct trampoline can co-exist together with graph tracer on same function.
This is supported for CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS config option, which is defined only for x86_64 for now.
--- kernel/trace/trace_selftest.c | 54 ++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-)
kernel/trace/trace_selftest.c +53 -1
@@ -750,6 +750,8 @@ static struct fgraph_ops fgraph_ops __initdata = { .retfunc = &trace_graph_return, }; +noinline __noclone static void trace_direct_tramp(void) { } +/* * Pretty much the same than for the function tracer from which the selftest * has been borrowed.
@@ -760,6 +762,7 @@ trace_selftest_startup_function_graph(struct tracer *trace, {int ret;unsigned long count; +char *func_name__maybe_unused ;#ifdef CONFIG_DYNAMIC_FTRACE if (ftrace_filter_param) {
@@ -808,8 +811,57 @@ trace_selftest_startup_function_graph(struct tracer *trace, goto out; } -/* Don't test dynamic tracing, the function tracer already did */ +#ifdef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_ARGS +tracing_reset_online_cpus (&tr->array_buffer); +set_graph_array (tr); + +/* +* Some archs *cough*PowerPC*cough* add characters to the +* start of the function names. We simply put a '*' to +* accommodate them. +*/ + func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); +ftrace_set_global_filter (func_name,strlen (func_name),1 ); + +/* +* Register direct function together with graph tracer +* and make sure we get graph trace. +*/ + ret =register_ftrace_direct ((unsigned long )DYN_FTRACE_TEST_NAME , + (unsigned long ) trace_direct_tramp); +if (ret) + goto out; + + ret =register_ftrace_graph (&fgraph_ops); +if (ret) { +warn_failed_init_tracer (trace, ret); + goto out; + } + +DYN_FTRACE_TEST_NAME (); + + count =0 ; + +tracing_stop (); +/* check the trace buffer */ + ret =trace_test_buffer (&tr->array_buffer, &count); + +unregister_ftrace_graph (&fgraph_ops); + + ret =unregister_ftrace_direct ((unsigned long )DYN_FTRACE_TEST_NAME , + (unsigned long ) trace_direct_tramp); +if (ret) + goto out; + +tracing_start (); +if (!ret && !count) { + ret =-1 ; + goto out; + } +#endif + +/* Don't test dynamic tracing, the function tracer already did */ out:/* Stop it if we failed */ if (ret)
Signature · 1 line
2.31.1
PATCH v2 5/8 ftrace: Add ftrace_add_rec_direct function
In reply to Jiri OlsaFactor out the code that adds (ip, addr) tuple to direct_functions hash in new ftrace_add_rec_direct function. It will be used in following patches.
--- kernel/trace/ftrace.c | 64 +++++++++++++++++++++++++------------------ 1 file changed, 37 insertions(+), 27 deletions(-)
kernel/trace/ftrace.c +37 -27
@@ -2394,6 +2394,39 @@ unsigned long ftrace_find_rec_direct(unsigned long ip)return entry->direct; } +static struct ftrace_func_entry * +ftrace_add_rec_direct (unsigned long ip,unsigned long addr, +struct ftrace_hash **free_hash) +{ +struct ftrace_func_entry *entry; + +if (ftrace_hash_empty (direct_functions) || + direct_functions->count >2 * (1 << direct_functions->size_bits)) { +struct ftrace_hash *new_hash; +int size =ftrace_hash_empty (direct_functions) ?0 : + direct_functions->count +1 ; + +if (size <32 ) + size =32 ; + + new_hash =dup_hash (direct_functions, size); +if (!new_hash) +return NULL ; + + *free_hash = direct_functions; + direct_functions = new_hash; + } + + entry =kmalloc (sizeof (*entry),GFP_KERNEL ); +if (!entry) +return NULL ; + + entry->ip = ip; + entry->direct = addr; +__add_hash_entry (direct_functions, entry); +return entry; +} +static void call_direct_funcs (unsigned long ip,unsigned long pip,struct ftrace_ops *ops,struct ftrace_regs *fregs) {
@@ -5110,39 +5143,16 @@ int register_ftrace_direct(unsigned long ip, unsigned long addr) } ret = -ENOMEM ; -if (ftrace_hash_empty (direct_functions) || - direct_functions->count >2 * (1 << direct_functions->size_bits)) { -struct ftrace_hash *new_hash; -int size =ftrace_hash_empty (direct_functions) ?0 : - direct_functions->count +1 ; - -if (size <32 ) - size =32 ; - - new_hash =dup_hash (direct_functions, size); -if (!new_hash) - goto out_unlock; - - free_hash = direct_functions; - direct_functions = new_hash; - } - - entry =kmalloc (sizeof (*entry),GFP_KERNEL ); -if (!entry) - goto out_unlock; - direct =ftrace_find_direct_func (addr);if (!direct) { direct =ftrace_alloc_direct_func (addr); -if (!direct) { -kfree (entry); +if (!direct) goto out_unlock; - } } - entry->ip = ip; - entry->direct = addr; -__add_hash_entry (direct_functions, entry); + entry =ftrace_add_rec_direct (ip, addr, &free_hash); +if (!entry) + goto out_unlock; ret =ftrace_set_filter_ip (&direct_ops, ip,0 ,0 );if (ret)
Signature · 1 line
2.31.1
PATCH v2 6/8 ftrace: Add multi direct register/unregister interface
In reply to Jiri OlsaAdding interface to register multiple direct functions within single call. Adding following functions:
register_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)
The register_ftrace_direct_multi registers direct function (addr) with all functions in ops filter. The ops filter can be updated before with ftrace_set_filter_ip calls.
All requested functions must not have direct function currently registered, otherwise register_ftrace_direct_multi will fail.
The unregister_ftrace_direct_multi unregisters ops related direct functions.
--- include/linux/ftrace.h | 11 ++++ kernel/trace/ftrace.c | 142 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+)
include/linux/ftrace.h +11 -0
@@ -316,7 +316,10 @@ int ftrace_modify_direct_caller(struct ftrace_func_entry *entry,unsigned long old_addr, unsigned long new_addr);unsigned long ftrace_find_rec_direct (unsigned long ip); +int register_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr); +int unregister_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr);#else +struct ftrace_ops ;# define ftrace_direct_func_count 0static inline int register_ftrace_direct (unsigned long ip,unsigned long addr) {
@@ -346,6 +349,14 @@ static inline unsigned long ftrace_find_rec_direct(unsigned long ip) {return 0 ; } +static inline int register_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) +{ +return -ENODEV ; +} +static inline int unregister_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) +{ +return -ENODEV ; +}#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
kernel/trace/ftrace.c +142 -0
148 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v2 7/8 ftrace: Add multi direct modify interface
In reply to Jiri OlsaAdding interface to modify registered direct function for ftrace_ops. Adding following function:
modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)
The function changes the currently registered direct function for all attached functions.
--- include/linux/ftrace.h | 6 ++++ kernel/trace/ftrace.c | 62 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 68 insertions(+)
include/linux/ftrace.h +6 -0
@@ -318,6 +318,8 @@ int ftrace_modify_direct_caller(struct ftrace_func_entry *entry,unsigned long ftrace_find_rec_direct (unsigned long ip);int register_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr);int unregister_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr); +int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr); +#else struct ftrace_ops ;# define ftrace_direct_func_count 0
@@ -357,6 +359,10 @@ static inline int unregister_ftrace_direct_multi(struct ftrace_ops *ops, unsigne {return -ENODEV ; } +static inline int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) +{ +return -ENODEV ; +}#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS */ #ifndef CONFIG_HAVE_DYNAMIC_FTRACE_WITH_DIRECT_CALLS
kernel/trace/ftrace.c +62 -0
68 lines not on this page · raw message
Signature · 1 line
2.31.1
PATCH v2 8/8 ftrace/samples: Add multi direct interface test module
In reply to Jiri OlsaAdding simple module that uses multi direct interface:
register_ftrace_direct_multi
unregister_ftrace_direct_multi
The init function registers trampoline for 2 functions, and exit function unregisters them.
--- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 samples/ftrace/ftrace-direct-multi.c
samples/ftrace/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o +obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-multi.o CFLAGS_sample-trace-array.o := -I$(src) obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += sample-trace-array.o
samples/ftrace/ftrace-direct-multi.c new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.31.1
Quoted 4 lines · Jiri Olsa
hi,
adding interface to maintain multiple direct functions within single calls. It's a base for follow up bpf batch attach functionality.
...
Quoted 13 lines
---
Jiri Olsa (6):
x86/ftrace: Remove extra orig rax move
tracing: Add trampoline/graph selftest
ftrace: Addftrace_add_rec_directfunction
ftrace: Add multi direct register/unregister interface
ftrace: Add multi direct modify interface ftrace/samples: Add multi direct interface test moduleSteven Rostedt (VMware) (2):
x86/ftrace: Remove fault protection code inprepare_ftrace_returnx86/ftrace: Make function graph use ftrace directly
Quoted 11 lines
arch/x86/include/asm/ftrace.h | 9 +++- arch/x86/kernel/ftrace.c | 71 +++++++++++++++--------------- arch/x86/kernel/ftrace_64.S | 30 +------------ include/linux/ftrace.h | 26 +++++++++++ kernel/trace/fgraph.c | 6 ++- kernel/trace/ftrace.c | 268 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------ kernel/trace/trace_selftest.c | 54 ++++++++++++++++++++++- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++ 9 files changed, 420 insertions(+), 97 deletions(-) create mode 100644 samples/ftrace/ftrace-direct-multi.c
FWIW, Steven pointed me to this thread since I posted DYNAMIC_FTRACE_WITH_DIRECT_CALL support for s390 here:
https://lore.kernel.org/all/[email protected]/
Since Jiri asked for it: please feel free to add
to all non-x86 patches.
Quoted 31 lines · Heiko Carstens
Quoted 4 lines · Jiri Olsa
hi,
adding interface to maintain multiple direct functions within single calls. It's a base for follow up bpf batch attach functionality....
Quoted 13 lines
---
Jiri Olsa (6):
x86/ftrace: Remove extra orig rax move
tracing: Add trampoline/graph selftest
ftrace: Addftrace_add_rec_directfunction
ftrace: Add multi direct register/unregister interface
ftrace: Add multi direct modify interface ftrace/samples: Add multi direct interface test moduleSteven Rostedt (VMware) (2):
x86/ftrace: Remove fault protection code inprepare_ftrace_returnx86/ftrace: Make function graph use ftrace directlyQuoted 11 lines
arch/x86/include/asm/ftrace.h | 9 +++- arch/x86/kernel/ftrace.c | 71 +++++++++++++++--------------- arch/x86/kernel/ftrace_64.S | 30 +------------ include/linux/ftrace.h | 26 +++++++++++ kernel/trace/fgraph.c | 6 ++- kernel/trace/ftrace.c | 268 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------ kernel/trace/trace_selftest.c | 54 ++++++++++++++++++++++- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++ 9 files changed, 420 insertions(+), 97 deletions(-) create mode 100644 samples/ftrace/ftrace-direct-multi.c
Quoted 8 lines
FWIW, Steven pointed me to this thread since I posted
DYNAMIC_FTRACE_WITH_DIRECT_CALLsupport for s390 here:
https://lore.kernel.org/all/[email protected]/Since Jiri asked for it: please feel free to add
Tested-by Heiko Carstens <[email protected]>to all non-x86 patches.
thanks,
jirka
Steven Rostedt
maintainerRe: [PATCH 7/8] ftrace: Add multi direct modify interface
In reply to Jiri OlsaReview
Quoted 12 lines · On Fri, 8 Oct 2021 11:13:35 +0200 Jiri Olsa <[email protected]> wrote:
(quoted diff) +12
-0+ /* + * Shutdown the ops, change 'direct' pointer for each + * ops entry in direct_functions hash and startup the + * ops back again. + * + * Note there is no callback called for @ops object after + * this ftrace_shutdown call until ftrace_startup is called + * later on. + */ + err = ftrace_shutdown(ops, 0); + if (err) + goto out_unlock;
I believe I said before that we can do this by adding a stub ops that match all the functions with the direct ops being modified. This will cause the loop function to be called, which will call the direct function helper, which will then call the direct function that is found. That is, there is no "pause" in calling the direct callers. Either the old direct is called, or the new one. When the function returns, all are calling the new one.
That is, instead of:
[ Changing direct call from my_direct_1 to my_direct_2 ]
<traced_func>:
call my_direct_1
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
nop
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
call my_direct_2
We have it do:
<traced_func>:
call my_direct_1
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
call ftrace_caller
<ftrace_caller>:
[..]
call ftrace_ops_list_funcftrace_ops_list_func()
{
ops->func() -> direct_helper -> set rax to my_direct_1 or my_direct_2
} call rax (to either my_direct_1 or my_direct_2
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
call my_direct_2
I did this on top of this patch:
--- kernel/trace/ftrace.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
kernel/trace/ftrace.c +20 -13
@@ -5561,8 +5561,12 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);*/ int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) { -struct ftrace_hash *hash = ops->func_hash->filter_hash; +struct ftrace_hash *hash;struct ftrace_func_entry *entry, *iter; +static struct ftrace_ops tmp_ops = { + .func = ftrace_stub, + .flags =FTRACE_OPS_FL_STUB , + };int i, size;int err;
@@ -5572,21 +5576,22 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)return -EINVAL ;mutex_lock (&direct_mutex); -mutex_lock (&ftrace_lock); + +/* Enable the tmp_ops to have the same functions as the direct ops */ +ftrace_ops_init (&tmp_ops); + tmp_ops.func_hash = ops->func_hash; + + err =register_ftrace_function (&tmp_ops); +if (err) + goto out_direct;/* -* Shutdown the ops, change 'direct' pointer for each -* ops entry in direct_functions hash and startup the -* ops back again. -* -* Note there is no callback called for @ops object after -* this ftrace_shutdown call until ftrace_startup is called -* later on. +* Now the ftrace_ops_list_func() is called to do the direct callers. +* We can safely change the direct functions attached to each entry. */ - err =ftrace_shutdown (ops,0 ); -if (err) - goto out_unlock; +mutex_lock (&ftrace_lock); + hash = ops->func_hash->filter_hash; size =1 << hash->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (iter, &hash->buckets[i], hlist) {
@@ -5597,10 +5602,12 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } } - err =ftrace_startup (ops,0 ); +/* Removing the tmp_ops will add the updated direct callers to the functions */ +unregister_ftrace_function (&tmp_ops); out_unlock:mutex_unlock (&ftrace_lock); + out_direct:mutex_unlock (&direct_mutex);return err; }
Signature · 1 line
2.31.1
Re: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 60 lines · Steven Rostedt
Review
Quoted 12 lines · On Fri, 8 Oct 2021 11:13:35 +0200 Jiri Olsa <[email protected]> wrote:
(quoted diff) +12
-0+ /* + * Shutdown the ops, change 'direct' pointer for each + * ops entry in direct_functions hash and startup the + * ops back again. + * + * Note there is no callback called for @ops object after + * this ftrace_shutdown call until ftrace_startup is called + * later on. + */ + err = ftrace_shutdown(ops, 0); + if (err) + goto out_unlock;I believe I said before that we can do this by adding a stub ops that match all the functions with the direct ops being modified. This will cause the loop function to be called, which will call the direct function helper, which will then call the direct function that is found. That is, there is no "pause" in calling the direct callers. Either the old direct is called, or the new one. When the function returns, all are calling the new one.
That is, instead of:
[ Changing direct call from
my_direct_1tomy_direct_2]<
traced_func>:
callmy_direct_1||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
nop||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callmy_direct_2We have it do:
<
traced_func>:
callmy_direct_1||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callftrace_caller<
ftrace_caller>:[..] call ftrace_ops_list_func
Quoted 6 lines
ftrace_ops_list_func(){ ops->func() -> direct_helper -> set rax to my_direct_1 or my_direct_2 }call rax (to either
my_direct_1ormy_direct_2
nice! :) I did not see that as a problem and something that can be done later, thanks for doing this
Quoted 9 lines
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callmy_direct_2I did this on top of this patch:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
thanks,
jirka
Quoted 75 lines
Signed-off-by Steven Rostedt (VMware) <[email protected]>--- kernel/trace/ftrace.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)kernel/trace/ftrace.c +20
-13@@ -5561,8 +5561,12 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);*/ int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) { -struct ftrace_hash *hash = ops->func_hash->filter_hash; +struct ftrace_hash *hash;struct ftrace_func_entry *entry, *iter; +static struct ftrace_ops tmp_ops = { + .func = ftrace_stub, + .flags =FTRACE_OPS_FL_STUB , + };int i, size;int err;@@ -5572,21 +5576,22 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)return -EINVAL ;mutex_lock (&direct_mutex); -mutex_lock (&ftrace_lock); + +/* Enable the tmp_ops to have the same functions as the direct ops */ +ftrace_ops_init (&tmp_ops); + tmp_ops.func_hash = ops->func_hash; + + err =register_ftrace_function (&tmp_ops); +if (err) + goto out_direct;/* -* Shutdown the ops, change 'direct' pointer for each -* ops entry in direct_functions hash and startup the -* ops back again. -* -* Note there is no callback called for @ops object after -* this ftrace_shutdown call until ftrace_startup is called -* later on. +* Now the ftrace_ops_list_func() is called to do the direct callers. +* We can safely change the direct functions attached to each entry. */ - err =ftrace_shutdown (ops,0 ); -if (err) - goto out_unlock; +mutex_lock (&ftrace_lock); + hash = ops->func_hash->filter_hash; size =1 << hash->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (iter, &hash->buckets[i], hlist) {@@ -5597,10 +5602,12 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } } - err =ftrace_startup (ops,0 ); +/* Removing the tmp_ops will add the updated direct callers to the functions */ +unregister_ftrace_function (&tmp_ops); out_unlock:mutex_unlock (&ftrace_lock); + out_direct:mutex_unlock (&direct_mutex);return err; }Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 2 lines · On Fri, 15 Oct 2021 14:05:25 +0200 Jiri Olsa <[email protected]> wrote:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
If you want, I'll apply this as a separate change on top of your patch set. As I don't see anything wrong with your current code.
And when you are satisfied with this, just give me a "tested-by" and I'll push it too.
-- Steve
Re: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 11 lines · Steven Rostedt
Quoted 2 lines · On Fri, 15 Oct 2021 14:05:25 +0200 Jiri Olsa <[email protected]> wrote:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
If you want, I'll apply this as a separate change on top of your patch set. As I don't see anything wrong with your current code.
And when you are satisfied with this, just give me a "tested-by" and I'll push it too.
sounds great, thanks
jirka
Steven Rostedt
maintainerRe: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 7 lines · On Sat, 16 Oct 2021 13:39:55 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 6 lines · Steven Rostedt
Quoted 2 lines · On Fri, 15 Oct 2021 14:05:25 +0200 Jiri Olsa <[email protected]> wrote:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
Quoted 8 lines
Quoted 5 lines
If you want, I'll apply this as a separate change on top of your patch set. As I don't see anything wrong with your current code.
And when you are satisfied with this, just give me a "tested-by" and I'll push it too.
sounds great, thanks
jirka
Would you want to ack/review this?
-- Steve
From: "Steven Rostedt (VMware)" <[email protected]>
Subject: [PATCH] ftrace/direct: Do not disable when switching direct callers
Currently to switch a set of "multi" direct trampolines from one trampoline to another, a full shutdown of the current set needs to be done, followed by an update to what trampoline the direct callers would call, and then re-enabling the callers. This leaves a time when the functions will not be calling anything, and events may be missed.
Instead, use a trick to allow all the functions with direct trampolines attached will always call either the new or old trampoline while the switch is happening. To do this, first attach a "dummy" callback via ftrace to all the functions that the current direct trampoline is attached to. This will cause the functions to call the "list func" instead of the direct trampoline. The list function will call the direct trampoline "helper" that will set the function it should call as it returns back to the ftrace trampoline.
At this moment, the direct caller descriptor can safely update the direct call trampoline. The list function will pick either the new or old function (depending on the memory coherency model of the architecture).
Now removing the dummy function from each of the locations of the direct trampoline caller, will put back the direct call, but now to the new trampoline.
A better visual is:
[ Changing direct call from my_direct_1 to my_direct_2 ]
<traced_func>:
call my_direct_1
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
call ftrace_caller
<ftrace_caller>:
[..]
call ftrace_ops_list_func
ftrace_ops_list_func()
{
ops->func() -> direct_helper -> set rax to my_direct_1 or my_direct_2
} call rax (to either my_direct_1 or my_direct_2
||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv
<traced_func>:
call my_direct_2
--- kernel/trace/ftrace.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)
kernel/trace/ftrace.c +20 -13
@@ -5561,8 +5561,12 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);*/ int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) { -struct ftrace_hash *hash = ops->func_hash->filter_hash; +struct ftrace_hash *hash;struct ftrace_func_entry *entry, *iter; +static struct ftrace_ops tmp_ops = { + .func = ftrace_stub, + .flags =FTRACE_OPS_FL_STUB , + };int i, size;int err;
@@ -5572,21 +5576,22 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)return -EINVAL ;mutex_lock (&direct_mutex); -mutex_lock (&ftrace_lock); + +/* Enable the tmp_ops to have the same functions as the direct ops */ +ftrace_ops_init (&tmp_ops); + tmp_ops.func_hash = ops->func_hash; + + err =register_ftrace_function (&tmp_ops); +if (err) + goto out_direct;/* -* Shutdown the ops, change 'direct' pointer for each -* ops entry in direct_functions hash and startup the -* ops back again. -* -* Note there is no callback called for @ops object after -* this ftrace_shutdown call until ftrace_startup is called -* later on. +* Now the ftrace_ops_list_func() is called to do the direct callers. +* We can safely change the direct functions attached to each entry. */ - err =ftrace_shutdown (ops,0 ); -if (err) - goto out_unlock; +mutex_lock (&ftrace_lock); + hash = ops->func_hash->filter_hash; size =1 << hash->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (iter, &hash->buckets[i], hlist) {
@@ -5597,10 +5602,12 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } } - err =ftrace_startup (ops,0 ); +/* Removing the tmp_ops will add the updated direct callers to the functions */ +unregister_ftrace_function (&tmp_ops); out_unlock:mutex_unlock (&ftrace_lock); + out_direct:mutex_unlock (&direct_mutex);return err; }
Signature · 1 line
2.31.1
Re: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 13 lines · Steven Rostedt
Quoted 10 lines · On Sat, 16 Oct 2021 13:39:55 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 9 lines · Steven Rostedt
Quoted 2 lines · On Fri, 15 Oct 2021 14:05:25 +0200 Jiri Olsa <[email protected]> wrote:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
If you want, I'll apply this as a separate change on top of your patch set. As I don't see anything wrong with your current code.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
And when you are satisfied with this, just give me a "tested-by" and I'll push it too.
sounds great, thanks
jirkaWould you want to ack/review this?
hum, do you have it in some branch already? I'm getting:
patching file kernel/trace/ftrace.c
Hunk #1 succeeded at 5521 with fuzz 1 (offset -40 lines).
Hunk #2 FAILED at 5576.
Hunk #3 succeeded at 5557 (offset -44 lines).
1 out of 3 hunks FAILED -- saving rejects to file kernel/trace/ftrace.c.rej
when trying to apply on top of my changes
thanks,
jirka
Quoted 135 lines
-- Steve
From: "Steven Rostedt (VMware)" <[email protected]>
Subject: [PATCH] ftrace/direct: Do not disable when switching direct callersCurrently to switch a set of "multi" direct trampolines from one trampoline to another, a full shutdown of the current set needs to be done, followed by an update to what trampoline the direct callers would call, and then re-enabling the callers. This leaves a time when the functions will not be calling anything, and events may be missed.
Instead, use a trick to allow all the functions with direct trampolines attached will always call either the new or old trampoline while the switch is happening. To do this, first attach a "dummy" callback via ftrace to all the functions that the current direct trampoline is attached to. This will cause the functions to call the "list func" instead of the direct trampoline. The list function will call the direct trampoline "helper" that will set the function it should call as it returns back to the ftrace trampoline.
At this moment, the direct caller descriptor can safely update the direct call trampoline. The list function will pick either the new or old function (depending on the memory coherency model of the architecture).
Now removing the dummy function from each of the locations of the direct trampoline caller, will put back the direct call, but now to the new trampoline.
A better visual is:
[ Changing direct call from
my_direct_1tomy_direct_2]<
traced_func>:
callmy_direct_1||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callftrace_caller<
ftrace_caller>:[..] call ftrace_ops_list_func ftrace_ops_list_func() { ops->func() -> direct_helper -> set rax to my_direct_1 or my_direct_2 }call rax (to either
my_direct_1ormy_direct_2||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callmy_direct_2Signed-off-by Steven Rostedt (VMware) <[email protected]>--- kernel/trace/ftrace.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)kernel/trace/ftrace.c +20
-13@@ -5561,8 +5561,12 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);*/ int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) { -struct ftrace_hash *hash = ops->func_hash->filter_hash; +struct ftrace_hash *hash;struct ftrace_func_entry *entry, *iter; +static struct ftrace_ops tmp_ops = { + .func = ftrace_stub, + .flags =FTRACE_OPS_FL_STUB , + };int i, size;int err;@@ -5572,21 +5576,22 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)return -EINVAL ;mutex_lock (&direct_mutex); -mutex_lock (&ftrace_lock); + +/* Enable the tmp_ops to have the same functions as the direct ops */ +ftrace_ops_init (&tmp_ops); + tmp_ops.func_hash = ops->func_hash; + + err =register_ftrace_function (&tmp_ops); +if (err) + goto out_direct;/* -* Shutdown the ops, change 'direct' pointer for each -* ops entry in direct_functions hash and startup the -* ops back again. -* -* Note there is no callback called for @ops object after -* this ftrace_shutdown call until ftrace_startup is called -* later on. +* Now the ftrace_ops_list_func() is called to do the direct callers. +* We can safely change the direct functions attached to each entry. */ - err =ftrace_shutdown (ops,0 ); -if (err) - goto out_unlock; +mutex_lock (&ftrace_lock); + hash = ops->func_hash->filter_hash; size =1 << hash->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (iter, &hash->buckets[i], hlist) {@@ -5597,10 +5602,12 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } } - err =ftrace_startup (ops,0 ); +/* Removing the tmp_ops will add the updated direct callers to the functions */ +unregister_ftrace_function (&tmp_ops); out_unlock:mutex_unlock (&ftrace_lock); + out_direct:mutex_unlock (&direct_mutex);return err; }Signature · 1 line
2.31.1
Re: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 24 lines · Jiri Olsa
Quoted 13 lines · Steven Rostedt
Quoted 10 lines · On Sat, 16 Oct 2021 13:39:55 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 9 lines · Steven Rostedt
Quoted 2 lines · On Fri, 15 Oct 2021 14:05:25 +0200 Jiri Olsa <[email protected]> wrote:
ATM I'm bit stuck on the bpf side of this whole change, I'll test it with my other changes when I unstuck myself ;-)
If you want, I'll apply this as a separate change on top of your patch set. As I don't see anything wrong with your current code.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
And when you are satisfied with this, just give me a "tested-by" and I'll push it too.
sounds great, thanks
jirkaWould you want to ack/review this?
hum, do you have it in some branch already? I'm getting:
Quoted 8 lines
patching file
kernel/trace/ftrace.c
Hunk #1 succeeded at 5521 with fuzz 1 (offset -40 lines).
Hunk #2 FAILED at 5576.
Hunk #3 succeeded at 5557 (offset -44 lines).
1 out of 3 hunks FAILED -- saving rejects to filekernel/trace/ftrace.c.rejwhen trying to apply on top of my changes
I updated my ftrace/direct branch, it actually still had the previous version.. sorry, perhaps this is the cause of fuzz
jirka
Quoted 139 lines
thanks,
jirkaQuoted 135 lines
-- Steve
From: "Steven Rostedt (VMware)" <[email protected]>
Subject: [PATCH] ftrace/direct: Do not disable when switching direct callersCurrently to switch a set of "multi" direct trampolines from one trampoline to another, a full shutdown of the current set needs to be done, followed by an update to what trampoline the direct callers would call, and then re-enabling the callers. This leaves a time when the functions will not be calling anything, and events may be missed.
Instead, use a trick to allow all the functions with direct trampolines attached will always call either the new or old trampoline while the switch is happening. To do this, first attach a "dummy" callback via ftrace to all the functions that the current direct trampoline is attached to. This will cause the functions to call the "list func" instead of the direct trampoline. The list function will call the direct trampoline "helper" that will set the function it should call as it returns back to the ftrace trampoline.
At this moment, the direct caller descriptor can safely update the direct call trampoline. The list function will pick either the new or old function (depending on the memory coherency model of the architecture).
Now removing the dummy function from each of the locations of the direct trampoline caller, will put back the direct call, but now to the new trampoline.
A better visual is:
[ Changing direct call from
my_direct_1tomy_direct_2]<
traced_func>:
callmy_direct_1||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callftrace_caller<
ftrace_caller>:[..] call ftrace_ops_list_func ftrace_ops_list_func() { ops->func() -> direct_helper -> set rax to my_direct_1 or my_direct_2 }call rax (to either
my_direct_1ormy_direct_2||||||||||||||||||||
vvvvvvvvvvvvvvvvvvvv<
traced_func>:
callmy_direct_2Signed-off-by Steven Rostedt (VMware) <[email protected]>--- kernel/trace/ftrace.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-)kernel/trace/ftrace.c +20
-13@@ -5561,8 +5561,12 @@ EXPORT_SYMBOL_GPL(unregister_ftrace_direct_multi);*/ int modify_ftrace_direct_multi (struct ftrace_ops *ops,unsigned long addr) { -struct ftrace_hash *hash = ops->func_hash->filter_hash; +struct ftrace_hash *hash;struct ftrace_func_entry *entry, *iter; +static struct ftrace_ops tmp_ops = { + .func = ftrace_stub, + .flags =FTRACE_OPS_FL_STUB , + };int i, size;int err;@@ -5572,21 +5576,22 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr)return -EINVAL ;mutex_lock (&direct_mutex); -mutex_lock (&ftrace_lock); + +/* Enable the tmp_ops to have the same functions as the direct ops */ +ftrace_ops_init (&tmp_ops); + tmp_ops.func_hash = ops->func_hash; + + err =register_ftrace_function (&tmp_ops); +if (err) + goto out_direct;/* -* Shutdown the ops, change 'direct' pointer for each -* ops entry in direct_functions hash and startup the -* ops back again. -* -* Note there is no callback called for @ops object after -* this ftrace_shutdown call until ftrace_startup is called -* later on. +* Now the ftrace_ops_list_func() is called to do the direct callers. +* We can safely change the direct functions attached to each entry. */ - err =ftrace_shutdown (ops,0 ); -if (err) - goto out_unlock; +mutex_lock (&ftrace_lock); + hash = ops->func_hash->filter_hash; size =1 << hash->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (iter, &hash->buckets[i], hlist) {@@ -5597,10 +5602,12 @@ int modify_ftrace_direct_multi(struct ftrace_ops *ops, unsigned long addr) } } - err =ftrace_startup (ops,0 ); +/* Removing the tmp_ops will add the updated direct callers to the functions */ +unregister_ftrace_function (&tmp_ops); out_unlock:mutex_unlock (&ftrace_lock); + out_direct:mutex_unlock (&direct_mutex);return err; }Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 4 lines · On Tue, 19 Oct 2021 15:26:21 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 1 line
when trying to apply on top of my changes
I updated my ftrace/direct branch, it actually still had the previous version.. sorry, perhaps this is the cause of fuzz
I just pushed it (including your patches) here:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
ftrace/core
This is where I keep my WIP code. It should not be used to base anything off of, as I rebase it constantly. But it has the current version I plan on testing.
You can make sure the patches in there have your latest version, as you can review my patch. I'll update the tags if you give me one.
-- Steve
Re: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 15 lines · Steven Rostedt
Quoted 4 lines · On Tue, 19 Oct 2021 15:26:21 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 1 line
when trying to apply on top of my changes
I updated my ftrace/direct branch, it actually still had the previous version.. sorry, perhaps this is the cause of fuzz
I just pushed it (including your patches) here:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.gitftrace/core
Quoted 6 lines
This is where I keep my WIP code. It should not be used to base anything off of, as I rebase it constantly. But it has the current version I plan on testing.
You can make sure the patches in there have your latest version, as you can review my patch. I'll update the tags if you give me one.
I'm getting error when compiling:
CC kernel/trace/ftrace.okernel/trace/ftrace.c: In function ‘modify_ftrace_direct_multi’:kernel/trace/ftrace.c:5608:2: error: label ‘out_unlock’ defined but not used [-Werror=unused-label]
5608 | out_unlock:
| ^~~~~~~~~~looks like out_unlock is nolonger needed, I removed it
jirka
Steven Rostedt
maintainerRe: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 8 lines · On Tue, 19 Oct 2021 16:03:03 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 2 lines
You can make sure the patches in there have your latest version, as you can review my patch. I'll update the tags if you give me one.
I'm getting error when compiling:
CC
kernel/trace/ftrace.okernel/trace/ftrace.c: In function ‘modify_ftrace_direct_multi’:kernel/trace/ftrace.c:5608:2: error: label ‘out_unlock’ defined but not used [-Werror=unused-label]
Ah, I don't think I've been hit by the "-Werror" yet ;-)
Quoted 4 lines
5608 | out_unlock: | ^~~~~~~~~~looks like
out_unlockis nolonger needed, I removed it
My tests would have found this, as it has a check for "new warnings".
Anyway, was this in your latest patch, or did I pull in and older one?
That is, should I expect a v2 from you?
-- Steve
Jiri Olsa
Acked-byRe: [PATCH 7/8] ftrace: Add multi direct modify interface
Quoted 18 lines · Steven Rostedt
Quoted 8 lines · On Tue, 19 Oct 2021 16:03:03 +0200 Jiri Olsa <[email protected]> wrote:
Quoted 2 lines
You can make sure the patches in there have your latest version, as you can review my patch. I'll update the tags if you give me one.
I'm getting error when compiling:
CC
kernel/trace/ftrace.okernel/trace/ftrace.c: In function ‘modify_ftrace_direct_multi’:kernel/trace/ftrace.c:5608:2: error: label ‘out_unlock’ defined but not used [-Werror=unused-label]Ah, I don't think I've been hit by the "-Werror" yet ;-)
Quoted 3 lines
5608 | out_unlock: | ^~~~~~~~~~
Quoted 7 lines
Quoted 1 line
looks like
out_unlockis nolonger needed, I removed itMy tests would have found this, as it has a check for "new warnings".
Anyway, was this in your latest patch, or did I pull in and older one?
That is, should I expect a v2 from you?
it's on top of your ftrace/core, in your change:
e62d91d8206e ftrace/direct: Do not disable when switching direct callers
just removing the label will fix it
also you can add my ack
thanks,
jirka
Re: [PATCH 8/8] ftrace/samples: Add multi direct interface test module
In reply to Jiri OlsaQuoted 9 lines · Jiri Olsa
Adding simple module that uses multi direct interface:
register_ftrace_direct_multi
unregister_ftrace_direct_multiThe init function registers trampoline for 2 functions, and exit function unregisters them.
Signed-off-by Jiri Olsa <[email protected]>
Building s390:defconfig ... failed
--------------
Error log:
<stdin>:1559:2: warning: #warning syscall futex_waitv not implemented [-Wcpp] {standard input}: Assembler messages:
{standard input}:11: Error: Unrecognized opcode: `pushq'
{standard input}:12: Error: Unrecognized opcode: `movq'
{standard input}:13: Error: Unrecognized opcode: `pushq'
{standard input}:14: Error: Unrecognized opcode: `movq'
{standard input}:15: Error: Unrecognized opcode: `call'
{standard input}:16: Error: Unrecognized opcode: `popq'
{standard input}:17: Error: Unrecognized opcode: `leave'
{standard input}:18: Error: Unrecognized opcode: `ret'
make[3]: *** [scripts/Makefile.build:288: samples/ftrace/ftrace-direct-multi.o] Error 1 make[2]: *** [scripts/Makefile.build:571: samples/ftrace] Error 2 make[1]: *** [Makefile:1993: samples] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:226: __sub-make] Error 2
Guenter
Quoted 79 lines
--- samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-direct-multi.c | 52 ++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+) create mode 100644 samples/ftrace/ftrace-direct-multi.csamples/ftrace/Makefile +1
-0@@ -3,6 +3,7 @@ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o +obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-multi.o CFLAGS_sample-trace-array.o := -I$(src) obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += sample-trace-array.osamples/ftrace/ftrace-direct-multi.c new file +52
-0@@ -0,0 +1,52 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/module.h> + +#include <linux/mm.h> /* for handle_mm_fault() */ +#include <linux/ftrace.h> +#include <linux/sched/stat.h> + +void my_direct_func (unsigned long ip) +{ +trace_printk ("ip %lx\n" , ip); +} + +extern void my_tramp (void *); + +asm ( +" .pushsection .text, \"ax\", @progbits\n" +" .type my_tramp, @function\n" +" .globl my_tramp\n" +" my_tramp:" +" pushq %rbp\n" +" movq %rsp, %rbp\n" +" pushq %rdi\n" +" movq 8(%rbp), %rdi\n" +" call my_direct_func\n" +" popq %rdi\n" +" leave\n" +" ret\n" +" .size my_tramp, .-my_tramp\n" +" .popsection\n" +); + +static struct ftrace_ops direct; + +static int __init ftrace_direct_multi_init (void ) +{ +ftrace_set_filter_ip (&direct, (unsigned long ) wake_up_process,0 ,0 ); +ftrace_set_filter_ip (&direct, (unsigned long ) schedule,0 ,0 ); + +return register_ftrace_direct_multi (&direct, (unsigned long ) my_tramp); +} + +static void __exit ftrace_direct_multi_exit(void) +{ +unregister_ftrace_direct_multi (&direct, (unsigned long ) my_tramp); +} + +module_init (ftrace_direct_multi_init); +module_exit (ftrace_direct_multi_exit); + +MODULE_AUTHOR ("Jiri Olsa" ); +MODULE_DESCRIPTION ("Example use case of using register_ftrace_direct_multi()" ); +MODULE_LICENSE ("GPL" );Signature · 1 line
2.31.1
Steven Rostedt
maintainerRe: [PATCH 8/8] ftrace/samples: Add multi direct interface test module
Quoted 21 lines · On Tue, 26 Oct 2021 12:23:09 -0700 Guenter Roeck <[email protected]> wrote:
Quoted 9 lines · Jiri Olsa
Adding simple module that uses multi direct interface:
register_ftrace_direct_multi
unregister_ftrace_direct_multiThe init function registers trampoline for 2 functions, and exit function unregisters them.
Signed-off-by Jiri Olsa <[email protected]>Building s390:defconfig ... failed
--------------
Error log:
<stdin>:1559:2: warning: #warning syscallfutex_waitvnot implemented [-Wcpp] {standard input}: Assembler messages:
{standard input}:11: Error: Unrecognized opcode:pushq' {standard input}:12: Error: Unrecognized opcode:movq'
{standard input}:13: Error: Unrecognized opcode:pushq' {standard input}:14: Error: Unrecognized opcode:movq'
{standard input}:15: Error: Unrecognized opcode: `call'
Quoted 10 lines
{standard input}:16: Error: Unrecognized opcode: `popq'
{standard input}:17: Error: Unrecognized opcode: `leave'
{standard input}:18: Error: Unrecognized opcode: `ret'
make[3]: *** [scripts/Makefile.build:288:samples/ftrace/ftrace-direct-multi.o] Error 1 make[2]: *** [scripts/Makefile.build:571:samples/ftrace] Error 2 make[1]: *** [Makefile:1993: samples] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:226: __sub-make] Error 2
https://lore.kernel.org/all/YXAqZ%2FEszRisunQw@osiris/
-- Steve