function_graph: Allow multiple users for function graph tracing
Steven Rostedt
maintainerv3 cover function_graph: Allow multiple users for function graph tracing
This is a continuation of the function graph multi user code. I wrote a proof of concept back in 2019 of this code[1] and Masami started cleaning it up. I started from Masami's work v10 that can be found here:
https://lore.kernel.org/linux-trace-kernel/171509088006.162236.7227326999861366050.stgit@devnote2/
This is *only* the code that allows multiple users of function graph tracing. This is not the fprobe work that Masami is working to add on top of it. As Masami took my proof of concept, there was still several things I disliked about that code. Instead of having Masami clean it up even more, I decided to take over on just my code and change it up a bit.
Changes since v2: https://lore.kernel.org/linux-trace-kernel/[email protected]
- Added comments describing which hashes the append and intersect functions were used for.
- Replaced checks of (NULL or EMPTY_HASH) with ftrace_hash_empty() helper function.
- Added check at the end of intersect_hash() to convert the hash to EMPTY hash if it doesn't have any functions.
- Renamed compare_ops() to ops_equal() and return boolean (inversed return value).
- Broke out __ftrace_hash_move_and_update_ops() to use in both ftrace_hash_move_and_update_ops() and ftrace_hash_move_and_update_subops().
Diff between last version at end of this email.
Masami Hiramatsu (Google) (3):
function_graph: Handle tail calls for stack unwinding
function_graph: Use a simple LRU for fgraph_array index number
ftrace: Add multiple fgraph storage selftest
Steven Rostedt (Google) (9):
ftrace: Add subops logic to allow one ops to manage many
ftrace: Allow subops filtering to be modified
function_graph: Add pid tracing back to function graph tracer
function_graph: Use for_each_set_bit() in __ftrace_return_to_handler()
function_graph: Use bitmask to loop on fgraph entry
function_graph: Use static_call and branch to optimize entry function
function_graph: Use static_call and branch to optimize return function selftests/ftrace: Add function_graph tracer to func-filter-pid test selftests/ftrace: Add fgraph-multi.tc test
Steven Rostedt (VMware) (15):
function_graph: Convert ret_stack to a series of longs
fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic around ftrace_graph_entry and return ftrace/function_graph: Pass fgraph_ops to function graph callbacks
ftrace: Allow function_graph tracer to be enabled in instances
ftrace: Allow ftrace startup flags to exist without dynamic ftrace
function_graph: Have the instances use their own ftrace_ops for filtering
function_graph: Add "task variables" per task for fgraph_ops
function_graph: Move set_graph_function tests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data()
function_graph: Add selftest for passing local variables
----
include/linux/ftrace.h | 43 +- include/linux/sched.h | 2 +- include/linux/trace_recursion.h | 39 - kernel/trace/fgraph.c | 1044 ++++++++++++++++---- kernel/trace/ftrace.c | 522 +++++++++- kernel/trace/ftrace_internal.h | 5 +- kernel/trace/trace.h | 94 +- kernel/trace/trace_functions.c | 8 + kernel/trace/trace_functions_graph.c | 96 +- kernel/trace/trace_irqsoff.c | 10 +- kernel/trace/trace_sched_wakeup.c | 10 +- kernel/trace/trace_selftest.c | 259 ++++- .../selftests/ftrace/test.d/ftrace/fgraph-multi.tc | 103 ++ .../ftrace/test.d/ftrace/func-filter-pid.tc | 27 +- 14 files changed, 1945 insertions(+), 317 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tc
kernel/trace/ftrace.c +64 -46
@@ -3170,7 +3170,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command)/* Simply make a copy of @src and return it */ static struct ftrace_hash *copy_hash (struct ftrace_hash *src) { -if (!src || src ==EMPTY_HASH ) +if (ftrace_hash_empty (src))return EMPTY_HASH ;return alloc_and_copy_ftrace_hash (src->size_bits, src);
@@ -3187,6 +3187,9 @@ static struct ftrace_hash *copy_hash(struct ftrace_hash *src)* * Otherwise, go through all of @new_hash and add anything that @hash * doesn't already have, to @hash. +* +* The filter_hash updates uses just the append_hash() function +* and the notrace_hash does not. */ static int append_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash) {
@@ -3195,11 +3198,11 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)int i;/* An empty hash does everything */ -if (!*hash || *hash ==EMPTY_HASH ) +if (ftrace_hash_empty (*hash))return 0 ;/* If new_hash has everything make hash have everything */ -if (!new_hash || new_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;
@@ -3217,7 +3220,12 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)return 0 ; } -/* Add to @hash only those that are in both @new_hash1 and @new_hash2 */ +/* +* Add to @hash only those that are in both @new_hash1 and @new_hash2 +* +* The notrace_hash updates uses just the intersect_hash() function +* and the filter_hash does not. +*/ static int intersect_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash1,struct ftrace_hash *new_hash2) {
@@ -3229,8 +3237,7 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_has* If new_hash1 or new_hash2 is the EMPTY_HASH then make the hash * empty as well as empty for notrace means none are notraced. */ -if (!new_hash1 || new_hash1 ==EMPTY_HASH || - !new_hash2 || new_hash2 ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash1) ||ftrace_hash_empty (new_hash2)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;
@@ -3245,6 +3252,11 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hasreturn -ENOMEM ; } } +/* If nothing intersects, make it the empty set */ +if (ftrace_hash_empty (*hash)) { +free_ftrace_hash (*hash); + *hash =EMPTY_HASH ; + }return 0 ; }
@@ -3266,7 +3278,7 @@ static struct ftrace_hash *append_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash;
@@ -3300,59 +3312,76 @@ static struct ftrace_hash *intersect_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash; } -/* Returns 0 on equal or non-zero on non-equal */ -static int compare_ops (struct ftrace_hash *A ,struct ftrace_hash *B ) +static bool ops_equal (struct ftrace_hash *A ,struct ftrace_hash *B ) {struct ftrace_func_entry *entry;int size;int i; -if (!A ||A ==EMPTY_HASH ) -return !(!B ||B ==EMPTY_HASH ); +if (ftrace_hash_empty (A )) +return ftrace_hash_empty (B ); -if (!B ||B ==EMPTY_HASH ) -return !(!A ||A ==EMPTY_HASH ); +if (ftrace_hash_empty (B )) +return ftrace_hash_empty (A );if (A ->count !=B ->count) -return 1 ; +return false; size =1 <<A ->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (entry, &A ->buckets[i], hlist) {if (!__ftrace_lookup_ip (B , entry->ip)) -return 1 ; +return false; } } -return 0 ; +return true; } -static int ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, -struct ftrace_hash **orig_hash, -struct ftrace_hash *hash, -int enable); +static void ftrace_ops_update_code (struct ftrace_ops *ops, +struct ftrace_ops_hash *old_hash); + +static int __ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, +struct ftrace_hash **orig_hash, +struct ftrace_hash *hash, +int enable) +{ +struct ftrace_ops_hash old_hash_ops; +struct ftrace_hash *old_hash; +int ret; + + old_hash = *orig_hash; + old_hash_ops.filter_hash = ops->func_hash->filter_hash; + old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; + ret =ftrace_hash_move (ops, enable, orig_hash, hash); +if (!ret) { +ftrace_ops_update_code (ops, &old_hash_ops); +free_ftrace_hash_rcu (old_hash); + } +return ret; +}static int ftrace_update_ops (struct ftrace_ops *ops,struct ftrace_hash *filter_hash,struct ftrace_hash *notrace_hash) {int ret; -if (compare_ops (filter_hash, ops->func_hash->filter_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, - filter_hash,1 ); +if (!ops_equal (filter_hash, ops->func_hash->filter_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, + filter_hash,1 );if (ret <0 )return ret; } -if (compare_ops (notrace_hash, ops->func_hash->notrace_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, - notrace_hash,0 ); +if (!ops_equal (notrace_hash, ops->func_hash->notrace_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, + notrace_hash,0 );if (ret <0 )return ret; }
@@ -3438,8 +3467,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int* o If either notrace_hash is empty then the final stays empty * o Otherwise, the final is an intersection between the hashes */ -if (ops->func_hash->filter_hash ==EMPTY_HASH || - subops->func_hash->filter_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->filter_hash) || +ftrace_hash_empty (subops->func_hash->filter_hash)) { filter_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,
@@ -3454,8 +3483,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int } } -if (ops->func_hash->notrace_hash ==EMPTY_HASH || - subops->func_hash->notrace_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->notrace_hash) || +ftrace_hash_empty (subops->func_hash->notrace_hash)) { notrace_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,
@@ -3591,7 +3620,7 @@ static int ftrace_hash_move_and_update_subops(struct ftrace_ops *subops, }/* Move the hash over to the new hash */ - ret =ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable); + ret =__ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable);free_ftrace_hash (new_hash);
@@ -4822,11 +4851,6 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, struct ftrace_hash *hash, int enable) { -struct ftrace_ops_hash old_hash_ops; -struct ftrace_hash *old_hash; -struct ftrace_ops *op; -int ret; -if (ops->flags &FTRACE_OPS_FL_SUBOP )return ftrace_hash_move_and_update_subops (ops, orig_hash, hash, enable);
@@ -4838,6 +4862,8 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops,* it will not affect subops that share it. */ if (!(ops->flags &FTRACE_OPS_FL_ENABLED )) { +struct ftrace_ops *op; +/* Check if any other manager subops maps to this hash */ do_for_each_ftrace_op (op, ftrace_ops_list) {struct ftrace_ops *subops;
@@ -4851,15 +4877,7 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, }while_for_each_ftrace_op (op); } - old_hash = *orig_hash; - old_hash_ops.filter_hash = ops->func_hash->filter_hash; - old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; - ret =ftrace_hash_move (ops, enable, orig_hash, hash); -if (!ret) { -ftrace_ops_update_code (ops, &old_hash_ops); -free_ftrace_hash_rcu (old_hash); - } -return ret; +return __ftrace_hash_move_and_update_ops (ops, orig_hash, hash, enable); }static bool module_exists (const char *module)
Steven Rostedt
maintainerPATCH v3 1/27 function_graph: Convert ret_stack to a series of longs
From: "Steven Rostedt (VMware)" <[email protected]>
In order to make it possible to have multiple callbacks registered with the function_graph tracer, the retstack needs to be converted from an array of ftrace_ret_stack structures to an array of longs. This will allow to store the list of callbacks on the stack for the return side of the functions.
--- include/linux/sched.h | 2 +- kernel/trace/fgraph.c | 136 +++++++++++++++++++++++++----------------- 2 files changed, 83 insertions(+), 55 deletions(-)
include/linux/sched.h +1 -1
@@ -1402,7 +1402,7 @@ struct task_struct {int curr_ret_depth;/* Stack of return addresses for return function tracing: */ -struct ftrace_ret_stack *ret_stack; +unsigned long *ret_stack;/* Timestamp for last schedule: */ unsigned long long ftrace_timestamp;
kernel/trace/fgraph.c +82 -54
319 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 2/27 fgraph: Use BUILD_BUG_ON() to make sure we have structures divisible by long
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Instead of using "ALIGN()", use BUILD_BUG_ON() as the structures should always be divisible by sizeof(long).
Co-developed with Masami Hiramatsu:
--- kernel/trace/fgraph.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/trace/fgraph.c +5 -1
@@ -33,7 +33,7 @@* SHADOW_STACK_MAX_OFFSET: The max offset of the stack for a new frame to be added */ #define FGRAPH_FRAME_SIZE sizeof(struct ftrace_ret_stack) -#define FGRAPH_FRAME_OFFSET (ALIGN(FGRAPH_FRAME_SIZE, sizeof(long)) / sizeof(long)) +#define FGRAPH_FRAME_OFFSET DIV_ROUND_UP(FGRAPH_FRAME_SIZE, sizeof(long))#define SHADOW_STACK_SIZE (PAGE_SIZE)#define SHADOW_STACK_OFFSET \ (ALIGN(SHADOW_STACK_SIZE, sizeof(long)) / sizeof(long))
@@ -103,6 +103,8 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func,if (!current->ret_stack)return -EBUSY ; +BUILD_BUG_ON (SHADOW_STACK_SIZE %sizeof (long )); +/* * We must make sure the ret_stack is tested before we read * anything else.
@@ -326,6 +328,8 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int idx) {int index = task->curr_ret_stack; +BUILD_BUG_ON (FGRAPH_FRAME_SIZE %sizeof (long )); + index -=FGRAPH_FRAME_OFFSET * (idx +1 );if (index <0 )return NULL ;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 3/27 function_graph: Add an array structure that will allow multiple callbacks
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Add an array structure that will eventually allow the function graph tracer to have up to 16 simultaneous callbacks attached. It's an array of 16 fgraph_ops pointers, that is assigned when one is registered. On entry of a function the entry of the first item in the array is called, and if it returns zero, then the callback returns non zero if it wants the return callback to be called on exit of the function.
The array will simplify the process of having more than one callback attached to the same function, as its index into the array can be stored on the shadow stack. We need to only save the index, because this will allow the fgraph_ops to be freed before the function returns (which may happen if the function call schedule for a long time).
Co-developed with Masami Hiramatsu:
--- kernel/trace/fgraph.c | 114 ++++++++++++++++++++++++++++++------------ 1 file changed, 81 insertions(+), 33 deletions(-)
kernel/trace/fgraph.c +81 -33
@@ -52,6 +52,11 @@DEFINE_STATIC_KEY_FALSE (kill_ftrace_graph);int ftrace_graph_active; +static int fgraph_array_cnt; +#define FGRAPH_ARRAY_SIZE 16 + +static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE ]; +/* Both enabled by default (can be cleared by function_graph tracer flags */ static bool fgraph_sleep_time = true;
@@ -75,6 +80,20 @@ int __weak ftrace_disable_ftrace_graph_caller(void) }#endif +int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) +{ +return 0 ; +} + +static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace) +{ +} + +static struct fgraph_ops fgraph_stub = { + .entryfunc = ftrace_graph_entry_stub, + .retfunc = ftrace_graph_ret_stub, +}; +/** * ftrace_graph_stop - set to permanently disable function graph tracing *
@@ -161,7 +180,7 @@ int function_graph_enter(unsigned long ret, unsigned long func, goto out;/* Only trace if the calling function expects to */ -if (!ftrace_graph_entry (&trace)) +if (!fgraph_array[0 ]->entryfunc (&trace)) goto out_ret;return 0 ;
@@ -276,7 +295,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs trace.retval =fgraph_ret_regs_return_value (ret_regs);#endif trace.rettime =trace_clock_local (); -ftrace_graph_return (&trace); + fgraph_array[0 ]->retfunc (&trace);/* * The ftrace_graph_return() may still access the current * ret_stack structure, we need to make sure the update of
@@ -412,11 +431,6 @@ void ftrace_graph_sleep_time_control(bool enable) fgraph_sleep_time = enable; } -int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) -{ -return 0 ; -} -/* * Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h
@@ -654,37 +668,54 @@ static int start_graph_tracing(void)int register_ftrace_graph (struct fgraph_ops *gops) {int ret =0 ; +int i;mutex_lock (&ftrace_lock); -/* we currently allow only one tracer registered at a time */ -if (ftrace_graph_active) { +if (!fgraph_array[0 ]) { +/* The array must always have real data on it */ +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) + fgraph_array[i] = &fgraph_stub; + } + +/* Look for an available spot */ +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { +if (fgraph_array[i] == &fgraph_stub) +break ; + } +if (i >=FGRAPH_ARRAY_SIZE ) { ret = -EBUSY ; goto out; } -register_pm_notifier (&ftrace_suspend_notifier); + fgraph_array[i] = gops; +if (i +1 > fgraph_array_cnt) + fgraph_array_cnt = i +1 ; ftrace_graph_active++; - ret =start_graph_tracing (); -if (ret) { - ftrace_graph_active--; - goto out; - } - ftrace_graph_return = gops->retfunc; +if (ftrace_graph_active ==1 ) { +register_pm_notifier (&ftrace_suspend_notifier); + ret =start_graph_tracing (); +if (ret) { + ftrace_graph_active--; + goto out; + } + + ftrace_graph_return = gops->retfunc; -/* -* Update the indirect function to the entryfunc, and the -* function that gets called to the entry_test first. Then -* call the update fgraph entry function to determine if -* the entryfunc should be called directly or not. -*/ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; -update_function_graph_func (); +/* +* Update the indirect function to the entryfunc, and the +* function that gets called to the entry_test first. Then +* call the update fgraph entry function to determine if +* the entryfunc should be called directly or not. +*/ + __ftrace_graph_entry = gops->entryfunc; + ftrace_graph_entry = ftrace_graph_entry_test; +update_function_graph_func (); - ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); + ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); + } out:mutex_unlock (&ftrace_lock);return ret;
@@ -692,19 +723,36 @@ int register_ftrace_graph(struct fgraph_ops *gops)void unregister_ftrace_graph (struct fgraph_ops *gops) { +int i; +mutex_lock (&ftrace_lock);if (unlikely (!ftrace_graph_active)) goto out; - ftrace_graph_active--; - ftrace_graph_return = ftrace_stub_graph; - ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub; -ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET ); -unregister_pm_notifier (&ftrace_suspend_notifier); -unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL ); +for (i =0 ; i < fgraph_array_cnt; i++) +if (gops == fgraph_array[i]) +break ; +if (i >= fgraph_array_cnt) + goto out; + fgraph_array[i] = &fgraph_stub; +if (i +1 == fgraph_array_cnt) { +for (; i >=0 ; i--) +if (fgraph_array[i] != &fgraph_stub) +break ; + fgraph_array_cnt = i +1 ; + } + + ftrace_graph_active--; +if (!ftrace_graph_active) { + ftrace_graph_return = ftrace_stub_graph; + ftrace_graph_entry = ftrace_graph_entry_stub; + __ftrace_graph_entry = ftrace_graph_entry_stub; +ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET ); +unregister_pm_notifier (&ftrace_suspend_notifier); +unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL ); + } out:mutex_unlock (&ftrace_lock); }
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 5/27 function_graph: Handle tail calls for stack unwinding
In reply to Steven RostedtFrom: "Masami Hiramatsu (Google)" <[email protected]>
For the tail-call, there would be 2 or more ftrace_ret_stacks on the ret_stack, which records "return_to_handler" as the return address except for the last one. But on the real stack, there should be 1 entry because tail-call reuses the return address on the stack and jump to the next function.
In ftrace_graph_ret_addr() that is used for stack unwinding, skip tail calls as a real stack unwinder would do.
--- kernel/trace/fgraph.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
kernel/trace/fgraph.c +16 -3
@@ -594,16 +594,26 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { struct ftrace_ret_stack *ret_stack; +unsigned long return_handler = (unsigned long )dereference_kernel_function_descriptor (return_to_handler);int i = task->curr_ret_stack; -if (ret != (unsigned long )dereference_kernel_function_descriptor (return_to_handler)) +if (ret != return_handler)return ret;while (i >0 ) { ret_stack =get_ret_stack (current, i, &i);if (!ret_stack)break ; -if (ret_stack->retp == retp) +/* +* For the tail-call, there would be 2 or more ftrace_ret_stacks on +* the ret_stack, which records "return_to_handler" as the return +* address except for the last one. +* But on the real stack, there should be 1 entry because tail-call +* reuses the return address on the stack and jump to the next function. +* Thus we will continue to find real return address. +*/ +if (ret_stack->retp == retp && + ret_stack->ret != return_handler)return ret_stack->ret; }
@@ -614,10 +624,11 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, unsigned long ret, unsigned long *retp) { struct ftrace_ret_stack *ret_stack; +unsigned long return_handler = (unsigned long )dereference_kernel_function_descriptor (return_to_handler);int offset = task->curr_ret_stack;int i; -if (ret != (unsigned long )dereference_kernel_function_descriptor (return_to_handler)) +if (ret != return_handler)return ret;if (!idx)
@@ -626,6 +637,8 @@ unsigned long ftrace_graph_ret_addr(struct task_struct *task, int *idx, i = *idx;do { ret_stack =get_ret_stack (task, offset, &offset); +if (ret_stack && ret_stack->ret == return_handler) +continue ; i--; }while (i >=0 && ret_stack);
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 4/27 function_graph: Allow multiple users to attach to function graph
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Allow for multiple users to attach to function graph tracer at the same time. Only 16 simultaneous users can attach to the tracer. This is because there's an array that stores the pointers to the attached fgraph_ops. When a function being traced is entered, each of the ftrace_ops entryfunc is called and if it returns non zero, its index into the array will be added to the shadow stack.
On exit of the function being traced, the shadow stack will contain the indexes of the ftrace_ops on the array that want their retfunc to be called.
Because a function may sleep for a long time (if a task sleeps itself), the return of the function may be literally days later. If the ftrace_ops is removed, its place on the array is replaced with a ftrace_ops that contains the stub functions and that will be called when the function finally returns.
If another ftrace_ops is added that happens to get the same index into the array, its return function may be called. But that's actually the way things current work with the old function graph tracer. If one tracer is removed and another is added, the new one will get the return calls of the function traced by the previous one, thus this is not a regression. This can be fixed by adding a counter to each time the array item is updated and save that on the shadow stack as well, such that it won't be called if the index saved does not match the index on the array.
Note, being able to filter functions when both are called is not completely handled yet, but that shouldn't be too hard to manage.
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 3 +- kernel/trace/fgraph.c | 379 ++++++++++++++++++++++++++++++++--------- 2 files changed, 304 insertions(+), 78 deletions(-)
include/linux/ftrace.h +2 -1
@@ -1038,6 +1038,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace);struct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +int idx; };/*
@@ -1072,7 +1073,7 @@ function_graph_enter(unsigned long ret, unsigned long func, unsigned long frame_pointer, unsigned long *retp);struct ftrace_ret_stack * -ftrace_graph_get_ret_stack (struct task_struct *task,int idx); +ftrace_graph_get_ret_stack (struct task_struct *task,int skip);unsigned long ftrace_graph_ret_addr (struct task_struct *task,int *idx,unsigned long ret,unsigned long *retp);
kernel/trace/fgraph.c +302 -77
578 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 6/27 function_graph: Remove logic around ftrace_graph_entry and return
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
The function pointers ftrace_graph_entry and ftrace_graph_return are no longer called via the function_graph tracer. Instead, an array structure is now used that will allow for multiple users of the function_graph infrastructure. The variables are still used by the architecture code for non dynamic ftrace configs, where a test is made against them to see if they point to the default stub function or not. This is how the static function tracing knows to call into the function graph tracer infrastructure or not.
Two new stub functions are made. entry_run() and return_run(). The ftrace_graph_entry and ftrace_graph_return are set to them respectively when the function graph tracer is enabled, and this will trigger the architecture specific function graph code to be executed.
This also requires checking the global_ops hash for all calls into the function_graph tracer.
Co-developed with Masami Hiramatsu:
--- kernel/trace/fgraph.c | 67 ++++++++-------------------------- kernel/trace/ftrace.c | 2 - kernel/trace/ftrace_internal.h | 2 - 3 files changed, 15 insertions(+), 56 deletions(-)
kernel/trace/fgraph.c +15 -52
@@ -163,6 +163,17 @@ set_bitmap(struct task_struct *t, int offset, unsigned long bitmap) (FGRAPH_TYPE_BITMAP <<FGRAPH_TYPE_SHIFT ) |FGRAPH_FRAME_OFFSET ; } +/* ftrace_graph_entry set to this to tell some archs to run function graph */ +static int entry_run (struct ftrace_graph_ent *trace) +{ +return 0 ; +} + +/* ftrace_graph_return set to this to tell some archs to run function graph */ +static void return_run (struct ftrace_graph_ret *trace) +{ +} +/* * @offset: The offset into @t->ret_stack to find the ret_stack entry * @frame_offset: Where to place the offset into @t->ret_stack of that entry
@@ -675,7 +686,6 @@ extern void ftrace_stub_graph(struct ftrace_graph_ret *);/* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub; -static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub;/* Try to assign a return stack array on FTRACE_RETSTACK_ALLOC_SIZE tasks. */ static int alloc_retstack_tasklist (unsigned long **ret_stack_list)
@@ -758,46 +768,6 @@ ftrace_graph_probe_sched_switch(void *ignore, bool preempt, } } -static int ftrace_graph_entry_test (struct ftrace_graph_ent *trace) -{ -if (!ftrace_ops_test (&global_ops, trace->func,NULL )) -return 0 ; -return __ftrace_graph_entry (trace); -} - -/* -* The function graph tracer should only trace the functions defined -* by set_ftrace_filter and set_ftrace_notrace. If another function -* tracer ops is registered, the graph tracer requires testing the -* function against the global ops, and not just trace any function -* that any ftrace_ops registered. -*/ -void update_function_graph_func (void ) -{ -struct ftrace_ops *op; -bool do_test = false; - -/* -* The graph and global ops share the same set of functions -* to test. If any other ops is on the list, then -* the graph tracing needs to test if its the function -* it should call. -*/ -do_for_each_ftrace_op (op, ftrace_ops_list) { -if (op != &global_ops && op != &graph_ops && - op != &ftrace_list_end) { - do_test = true; -/* in double loop, break out with goto */ - goto out; - } - }while_for_each_ftrace_op (op); - out: -if (do_test) - ftrace_graph_entry = ftrace_graph_entry_test; -else - ftrace_graph_entry = __ftrace_graph_entry; -} - static DEFINE_PER_CPU(unsigned long *, idle_ret_stack); static void
@@ -939,18 +909,12 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active--; goto out; } - - ftrace_graph_return = gops->retfunc; -/* -* Update the indirect function to the entryfunc, and the -* function that gets called to the entry_test first. Then -* call the update fgraph entry function to determine if -* the entryfunc should be called directly or not. +* Some archs just test to see if these are not +* the default function */ - __ftrace_graph_entry = gops->entryfunc; - ftrace_graph_entry = ftrace_graph_entry_test; -update_function_graph_func (); + ftrace_graph_return = return_run; + ftrace_graph_entry = entry_run; ret =ftrace_startup (&graph_ops,FTRACE_START_FUNC_RET ); }
@@ -986,7 +950,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops)if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub; - __ftrace_graph_entry = ftrace_graph_entry_stub;ftrace_shutdown (&graph_ops,FTRACE_STOP_FUNC_RET );unregister_pm_notifier (&ftrace_suspend_notifier);unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL );
kernel/trace/ftrace.c +0 -2
@@ -235,8 +235,6 @@ static void update_ftrace_function(void) func = ftrace_ops_list_func; } -update_function_graph_func (); -/* If there's no change, then do nothing more here */ if (ftrace_trace_function == func)return ;
kernel/trace/ftrace_internal.h +0 -2
@@ -42,10 +42,8 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs)#ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; -void update_function_graph_func (void );#else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 -static inline void update_function_graph_func (void ) { }#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 7/27 ftrace/function_graph: Pass fgraph_ops to function graph callbacks
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Pass the fgraph_ops structure to the function graph callbacks. This will allow callbacks to add a descriptor to a fgraph_ops private field that wil be added in the future and use it for the callbacks. This will be useful when more than one callback can be registered to the function graph tracer.
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 10 +++++++--- kernel/trace/fgraph.c | 16 +++++++++------- kernel/trace/ftrace.c | 6 ++++-- kernel/trace/trace.h | 4 ++-- kernel/trace/trace_functions_graph.c | 11 +++++++---- kernel/trace/trace_irqsoff.c | 6 ++++-- kernel/trace/trace_sched_wakeup.c | 6 ++++-- kernel/trace/trace_selftest.c | 5 +++-- 8 files changed, 40 insertions(+), 24 deletions(-)
include/linux/ftrace.h +7 -3
@@ -1027,11 +1027,15 @@ struct ftrace_graph_ret {unsigned long long rettime; } __packed; +struct fgraph_ops ; +/* Type of the callback handlers for tracing function graph*/ -typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *);/* return */ -typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *);/* entry */ +typedef void (*trace_func_graph_ret_t )(struct ftrace_graph_ret *, +struct fgraph_ops *);/* return */ +typedef int (*trace_func_graph_ent_t )(struct ftrace_graph_ent *, +struct fgraph_ops *);/* entry */ -extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace); +extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace,struct fgraph_ops *gops);#ifdef CONFIG_FUNCTION_GRAPH_TRACER
kernel/trace/fgraph.c +9 -7
@@ -164,13 +164,13 @@ set_bitmap(struct task_struct *t, int offset, unsigned long bitmap) }/* ftrace_graph_entry set to this to tell some archs to run function graph */ -static int entry_run (struct ftrace_graph_ent *trace) +static int entry_run (struct ftrace_graph_ent *trace,struct fgraph_ops *ops) {return 0 ; }/* ftrace_graph_return set to this to tell some archs to run function graph */ -static void return_run (struct ftrace_graph_ret *trace) +static void return_run (struct ftrace_graph_ret *trace,struct fgraph_ops *ops) { }
@@ -234,12 +234,14 @@ int __weak ftrace_disable_ftrace_graph_caller(void) }#endif -int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace) +int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {return 0 ; } -static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace) +static void ftrace_graph_ret_stub (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) { }
@@ -379,7 +381,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (gops == &fgraph_stub)continue ; -if (gops->entryfunc (&trace)) +if (gops->entryfunc (&trace, gops)) bitmap |=BIT (i); }
@@ -527,7 +529,7 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regsif (gops == &fgraph_stub)continue ; - gops->retfunc (&trace); + gops->retfunc (&trace, gops); }/*
@@ -681,7 +683,7 @@ void ftrace_graph_sleep_time_control(bool enable)* Simply points to ftrace_stub, but with the proper protocol. * Defined by the linker script in linux/vmlinux.lds.h */ -extern void ftrace_stub_graph (struct ftrace_graph_ret *); +void ftrace_stub_graph (struct ftrace_graph_ret *trace,struct fgraph_ops *gops);/* The callbacks that hook a function */ trace_func_graph_ret_t ftrace_graph_return = ftrace_stub_graph;
kernel/trace/ftrace.c +4 -2
@@ -815,7 +815,8 @@ void ftrace_graph_graph_time_control(bool enable) fgraph_graph_time = enable; } -static int profile_graph_entry (struct ftrace_graph_ent *trace) +static int profile_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct ftrace_ret_stack *ret_stack;
@@ -832,7 +833,8 @@ static int profile_graph_entry(struct ftrace_graph_ent *trace)return 1 ; } -static void profile_graph_return (struct ftrace_graph_ret *trace) +static void profile_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct ftrace_ret_stack *ret_stack;struct ftrace_profile_stat *stat;
kernel/trace/trace.h +2 -2
@@ -679,8 +679,8 @@ void trace_latency_header(struct seq_file *m);void trace_default_header (struct seq_file *m);void print_trace_header (struct seq_file *m,struct trace_iterator *iter); -void trace_graph_return (struct ftrace_graph_ret *trace); -int trace_graph_entry (struct ftrace_graph_ent *trace); +void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops); +int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops);void set_graph_array (struct trace_array *tr);void tracing_start_cmdline_record (void );
kernel/trace/trace_functions_graph.c +7 -4
@@ -129,7 +129,8 @@ static inline int ftrace_graph_ignore_irqs(void)return in_hardirq (); } -int trace_graph_entry (struct ftrace_graph_ent *trace) +int trace_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = graph_array;struct trace_array_cpu *data;
@@ -238,7 +239,8 @@ void __trace_graph_return(struct trace_array *tr,trace_buffer_unlock_commit_nostack (buffer, event); } -void trace_graph_return (struct ftrace_graph_ret *trace) +void trace_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = graph_array;struct trace_array_cpu *data;
@@ -275,7 +277,8 @@ void set_graph_array(struct trace_array *tr)smp_mb (); } -static void trace_graph_thresh_return (struct ftrace_graph_ret *trace) +static void trace_graph_thresh_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {ftrace_graph_addr_finish (trace);
@@ -288,7 +291,7 @@ static void trace_graph_thresh_return(struct ftrace_graph_ret *trace) (trace->rettime - trace->calltime < tracing_thresh)) return; else -trace_graph_return (trace ); +trace_graph_return (trace ,gops ); }static struct fgraph_ops funcgraph_thresh_ops = {
kernel/trace/trace_irqsoff.c +4 -2
@@ -175,7 +175,8 @@ static int irqsoff_display_graph(struct trace_array *tr, int set)return start_irqsoff_tracer (irqsoff_trace, set); } -static int irqsoff_graph_entry (struct ftrace_graph_ent *trace) +static int irqsoff_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
@@ -205,7 +206,8 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace)return ret; } -static void irqsoff_graph_return (struct ftrace_graph_ret *trace) +static void irqsoff_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = irqsoff_trace;struct trace_array_cpu *data;
kernel/trace/trace_sched_wakeup.c +4 -2
@@ -112,7 +112,8 @@ static int wakeup_display_graph(struct trace_array *tr, int set)return start_func_tracer (tr, set); } -static int wakeup_graph_entry (struct ftrace_graph_ent *trace) +static int wakeup_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
@@ -141,7 +142,8 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace)return ret; } -static void wakeup_graph_return (struct ftrace_graph_ret *trace) +static void wakeup_graph_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) {struct trace_array *tr = wakeup_trace;struct trace_array_cpu *data;
kernel/trace/trace_selftest.c +3 -2
@@ -762,7 +762,8 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)static unsigned int graph_hang_thresh;/* Wrap the real function entry probe to avoid possible hanging */ -static int trace_graph_entry_watchdog (struct ftrace_graph_ent *trace) +static int trace_graph_entry_watchdog (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) {/* This is harmlessly racy, we want to approximately detect a hang */ if (unlikely (++graph_hang_thresh >GRAPH_MAX_FUNC_TEST )) {
@@ -776,7 +777,7 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace)return 0 ; } -return trace_graph_entry (trace); +return trace_graph_entry (trace, gops); }static struct fgraph_ops fgraph_ops__initdata = {
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 8/27 ftrace: Allow function_graph tracer to be enabled in instances
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Now that function graph tracing can handle more than one user, allow it to be enabled in the ftrace instances. Note, the filtering of the functions is still joined by the top level set_ftrace_filter and friends, as well as the graph and nograph files.
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 1 + kernel/trace/ftrace.c | 1 + kernel/trace/trace.h | 13 +++++- kernel/trace/trace_functions.c | 8 ++++ kernel/trace/trace_functions_graph.c | 65 +++++++++++++++++----------- kernel/trace/trace_selftest.c | 4 +- 6 files changed, 64 insertions(+), 28 deletions(-)
include/linux/ftrace.h +1 -0
@@ -1042,6 +1042,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraphstruct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +void *private;int idx; };
kernel/trace/ftrace.c +1 -0
@@ -7327,6 +7327,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr;ftrace_init_trace_array (tr); +init_array_fgraph_ops (tr); }void ftrace_init_array_ops (struct trace_array *tr,ftrace_func_t func)
kernel/trace/trace.h +12 -1
@@ -397,6 +397,9 @@ struct trace_array {struct ftrace_ops *ops;struct trace_pid_list __rcu *function_pids;struct trace_pid_list __rcu *function_no_pids; +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +struct fgraph_ops *gops; +#endif #ifdef CONFIG_DYNAMIC_FTRACE /* All of these are protected by the ftrace_lock */ struct list_head func_probes;
@@ -681,7 +684,6 @@ void print_trace_header(struct seq_file *m, struct trace_iterator *iter);void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops);int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops); -void set_graph_array (struct trace_array *tr);void tracing_start_cmdline_record (void );void tracing_stop_cmdline_record (void );
@@ -892,6 +894,9 @@ extern int __trace_graph_entry(struct trace_array *tr,extern void __trace_graph_return (struct trace_array *tr,struct ftrace_graph_ret *trace,unsigned int trace_ctx); +extern void init_array_fgraph_ops (struct trace_array *tr); +extern int allocate_fgraph_ops (struct trace_array *tr); +extern void free_fgraph_ops (struct trace_array *tr);#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;
@@ -1004,6 +1009,12 @@ print_graph_function_flags(struct trace_iterator *iter, u32 flags) {return TRACE_TYPE_UNHANDLED ; } +static inline void init_array_fgraph_ops (struct trace_array *tr) { } +static inline int allocate_fgraph_ops (struct trace_array *tr) +{ +return 0 ; +} +static inline void free_fgraph_ops (struct trace_array *tr) { }#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids;
kernel/trace/trace_functions.c +8 -0
@@ -80,6 +80,7 @@ void ftrace_free_ftrace_ops(struct trace_array *tr)int ftrace_create_function_files (struct trace_array *tr,struct dentry *parent) { +int ret;/* * The top level array uses the "global_ops", and the files are * created on boot up.
@@ -90,6 +91,12 @@ int ftrace_create_function_files(struct trace_array *tr,if (!tr->ops)return -EINVAL ; + ret =allocate_fgraph_ops (tr); +if (ret) { +kfree (tr->ops); +return ret; + } +ftrace_create_filter_files (tr->ops, parent);return 0 ;
@@ -99,6 +106,7 @@ void ftrace_destroy_function_files(struct trace_array *tr) {ftrace_destroy_filter_files (tr->ops);ftrace_free_ftrace_ops (tr); +free_fgraph_ops (tr); }static ftrace_func_t select_trace_function (u32 flags_val)
kernel/trace/trace_functions_graph.c +40 -25
118 lines not on this page · raw message
kernel/trace/trace_selftest.c +2 -2
@@ -813,7 +813,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,* to detect and recover from possible hangs */ tracing_reset_online_cpus (&tr->array_buffer); -set_graph_array (tr); + fgraph_ops.private = tr; ret =register_ftrace_graph (&fgraph_ops);if (ret) {warn_failed_init_tracer (trace, ret);
@@ -856,7 +856,7 @@ trace_selftest_startup_function_graph(struct tracer *trace,cond_resched ();tracing_reset_online_cpus (&tr->array_buffer); -set_graph_array (tr); + fgraph_ops.private = tr;/* * Some archs *cough*PowerPC*cough* add characters to the
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 9/27 ftrace: Allow ftrace startup flags to exist without dynamic ftrace
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Some of the flags for ftrace_startup() may be exposed even when CONFIG_DYNAMIC_FTRACE is not configured in. This is fine as the difference between dynamic ftrace and static ftrace is done within the internals of ftrace itself. No need to have use cases fail to compile because dynamic ftrace is disabled.
This change is needed to move some of the logic of what is passed to ftrace_startup() out of the parameters of ftrace_startup().
--- include/linux/ftrace.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
include/linux/ftrace.h +9 -9
@@ -509,6 +509,15 @@ static inline void stack_tracer_disable(void) { }static inline void stack_tracer_enable (void ) { }#endif +enum { +FTRACE_UPDATE_CALLS = (1 <<0 ), +FTRACE_DISABLE_CALLS = (1 <<1 ), +FTRACE_UPDATE_TRACE_FUNC = (1 <<2 ), +FTRACE_START_FUNC_RET = (1 <<3 ), +FTRACE_STOP_FUNC_RET = (1 <<4 ), +FTRACE_MAY_SLEEP = (1 <<5 ), +}; +#ifdef CONFIG_DYNAMIC_FTRACE void ftrace_arch_code_modify_prepare (void );
@@ -603,15 +612,6 @@ void ftrace_set_global_notrace(unsigned char *buf, int len, int reset);void ftrace_free_filter (struct ftrace_ops *ops);void ftrace_ops_set_global_filter (struct ftrace_ops *ops); -enum { -FTRACE_UPDATE_CALLS = (1 <<0 ), -FTRACE_DISABLE_CALLS = (1 <<1 ), -FTRACE_UPDATE_TRACE_FUNC = (1 <<2 ), -FTRACE_START_FUNC_RET = (1 <<3 ), -FTRACE_STOP_FUNC_RET = (1 <<4 ), -FTRACE_MAY_SLEEP = (1 <<5 ), -}; -/* * The FTRACE_UPDATE_* enum is used to pass information back * from the ftrace_update_record() and ftrace_test_record()
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 10/27 ftrace: Add subops logic to allow one ops to manage many
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
There are cases where a single system will use a single function callback to handle multiple users. For example, to allow function_graph tracer to have multiple users where each can trace their own set of functions, it is useful to only have one ftrace_ops registered to ftrace that will call a function by the function_graph tracer to handle the multiplexing with the different registered function_graph tracers.
Add a "subop_list" to the ftrace_ops that will hold a list of other ftrace_ops that the top ftrace_ops will manage.
The function ftrace_startup_subops() that takes the manager ftrace_ops and a subop ftrace_ops it will manage. If there are no subops with the ftrace_ops yet, it will copy the ftrace_ops subop filters to the manager ftrace_ops and register that with ftrace_startup(), and adds the subop to its subop_list. If the manager ops already has something registered, it will then merge the new subop filters with what it has and enable the new functions that covers all the subops it has.
To remove a subop, ftrace_shutdown_subops() is called which will use the subop_list of the manager ops to rebuild all the functions it needs to trace, and update the ftrace records to only call the functions it now has registered. If there are no more functions registered, it will then call ftrace_shutdown() to disable itself completely.
Note, it is up to the manager ops callback to always make sure that the subops callbacks are called if its filter matches, as there are times in the update where the callback could be calling more functions than those that are currently registered.
This could be updated to handle other systems other than function_graph, for example, fprobes could use this (but will need an interface to call ftrace_startup_subops()).
--- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 3 +- kernel/trace/ftrace.c | 401 ++++++++++++++++++++++++++++++++- kernel/trace/ftrace_internal.h | 1 + kernel/trace/trace.h | 1 + 5 files changed, 405 insertions(+), 2 deletions(-)
include/linux/ftrace.h +1 -0
@@ -334,6 +334,7 @@ struct ftrace_ops {unsigned long trampoline;unsigned long trampoline_size;struct list_head list; +struct list_head subop_list;ftrace_ops_func_t ops_func;#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS unsigned long direct_call;
kernel/trace/fgraph.c +2 -1
@@ -21,7 +21,8 @@#ifdef CONFIG_DYNAMIC_FTRACE #define ASSIGN_OPS_HASH (opsname, val) \ .func_hash = val, \ - .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), + .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), \ + .subop_list = LIST_HEAD_INIT(opsname.subop_list),#else #define ASSIGN_OPS_HASH (opsname, val)#endif
kernel/trace/ftrace.c +400 -1
419 lines not on this page · raw message
kernel/trace/ftrace_internal.h +1 -0
@@ -15,6 +15,7 @@ extern struct ftrace_ops global_ops;int ftrace_startup (struct ftrace_ops *ops,int command);int ftrace_shutdown (struct ftrace_ops *ops,int command);int ftrace_ops_test (struct ftrace_ops *ops,unsigned long ip,void *regs); +int ftrace_shutdown_subops (struct ftrace_ops *ops,struct ftrace_ops *subops,int command);#else /* !CONFIG_DYNAMIC_FTRACE */
kernel/trace/trace.h +1 -0
@@ -1136,6 +1136,7 @@ extern int ftrace_set_filter(struct ftrace_ops *ops, unsigned char *buf, int len, int reset);extern int ftrace_set_notrace (struct ftrace_ops *ops,unsigned char *buf,int len,int reset); +extern int ftrace_startup_subops (struct ftrace_ops *ops,struct ftrace_ops *subops,int command);#else struct ftrace_func_command ;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 11/27 ftrace: Allow subops filtering to be modified
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The subops filters use a "manager" ops to enable and disable its filters. The manager ops can handle more than one subops, and its filter is what controls what functions get set. Add a ftrace_hash_move_and_update_subops() function that will update the manager ops when the subops filters change.
--- include/linux/ftrace.h | 3 + kernel/trace/ftrace.c | 141 +++++++++++++++++++++++++++++++++-------- 2 files changed, 116 insertions(+), 28 deletions(-)
include/linux/ftrace.h +3 -0
@@ -227,6 +227,7 @@ ftrace_func_t ftrace_ops_get_func(struct ftrace_ops *ops);* ftrace_enabled. * DIRECT - Used by the direct ftrace_ops helper for direct functions * (internal ftrace only, should not be used by others) +* SUBOP - Is controlled by another op in field managed. */ enum {FTRACE_OPS_FL_ENABLED =BIT (0 ),
@@ -247,6 +248,7 @@ enum {FTRACE_OPS_FL_TRACE_ARRAY =BIT (15 ),FTRACE_OPS_FL_PERMANENT =BIT (16 ),FTRACE_OPS_FL_DIRECT =BIT (17 ), + FTRACE_OPS_FL_SUBOP = BIT(18), };#ifndef CONFIG_DYNAMIC_FTRACE_WITH_ARGS
@@ -336,6 +338,7 @@ struct ftrace_ops {struct list_head list;struct list_head subop_list;ftrace_ops_func_t ops_func; +struct ftrace_ops *managed;#ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS unsigned long direct_call;#endif
kernel/trace/ftrace.c +113 -28
197 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 12/27 function_graph: Have the instances use their own ftrace_ops for filtering
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Allow for instances to have their own ftrace_ops part of the fgraph_ops that makes the funtion_graph tracer filter on the set_ftrace_filter file of the instance and not the top instance.
This uses the new ftrace_startup_subops(), by using graph_ops as the "manager ops" that defines the callback function and adds the functions defined by the filters of the ops for each trace instance. The callback defined by the manager ops will call the registered fgraph ops that were added to the fgraph_array.
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 81 +++++++++++++++++----------- kernel/trace/ftrace.c | 2 +- kernel/trace/trace.h | 15 +++--- kernel/trace/trace_functions.c | 2 +- kernel/trace/trace_functions_graph.c | 8 ++- 6 files changed, 68 insertions(+), 41 deletions(-)
include/linux/ftrace.h +1 -0
@@ -1046,6 +1046,7 @@ extern int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace, struct fgraphstruct fgraph_ops {trace_func_graph_ent_t entryfunc;trace_func_graph_ret_t retfunc; +struct ftrace_ops ops;/* for the hash lists */ void *private;int idx; };
kernel/trace/fgraph.c +51 -30
157 lines not on this page · raw message
kernel/trace/ftrace.c +1 -1
@@ -7811,7 +7811,7 @@ __init void ftrace_init_global_array_ops(struct trace_array *tr) tr->ops = &global_ops; tr->ops->private = tr;ftrace_init_trace_array (tr); -init_array_fgraph_ops (tr); +init_array_fgraph_ops (tr, tr->ops); }void ftrace_init_array_ops (struct trace_array *tr,ftrace_func_t func)
kernel/trace/trace.h +8 -7
@@ -894,8 +894,8 @@ extern int __trace_graph_entry(struct trace_array *tr,extern void __trace_graph_return (struct trace_array *tr,struct ftrace_graph_ret *trace,unsigned int trace_ctx); -extern void init_array_fgraph_ops (struct trace_array *tr); -extern int allocate_fgraph_ops (struct trace_array *tr); +extern void init_array_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops); +extern int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops);extern void free_fgraph_ops (struct trace_array *tr);#ifdef CONFIG_DYNAMIC_FTRACE
@@ -1003,18 +1003,19 @@ static inline bool ftrace_graph_ignore_func(struct ftrace_graph_ent *trace) (fgraph_max_depth && trace->depth >= fgraph_max_depth); } +void fgraph_init_ops (struct ftrace_ops *dst_ops, +struct ftrace_ops *src_ops); +#else /* CONFIG_FUNCTION_GRAPH_TRACER */ static inline enum print_line_t print_graph_function_flags (struct trace_iterator *iter,u32 flags) {return TRACE_TYPE_UNHANDLED ; } -static inline void init_array_fgraph_ops (struct trace_array *tr) { } -static inline int allocate_fgraph_ops (struct trace_array *tr) -{ -return 0 ; -}static inline void free_fgraph_ops (struct trace_array *tr) { } +/* ftrace_ops may not be defined */ +#define init_array_fgraph_ops (tr, ops) do { } while (0) +#define allocate_fgraph_ops (tr, ops) ({ 0; })#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ extern struct list_head ftrace_pids;
kernel/trace/trace_functions.c +1 -1
@@ -91,7 +91,7 @@ int ftrace_create_function_files(struct trace_array *tr,if (!tr->ops)return -EINVAL ; - ret =allocate_fgraph_ops (tr); + ret =allocate_fgraph_ops (tr, tr->ops);if (ret) {kfree (tr->ops);return ret;
kernel/trace/trace_functions_graph.c +6 -2
@@ -288,7 +288,7 @@ static struct fgraph_ops funcgraph_ops = { .retfunc = &trace_graph_return, }; -int allocate_fgraph_ops (struct trace_array *tr) +int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops) {struct fgraph_ops *gops;
@@ -301,6 +301,9 @@ int allocate_fgraph_ops(struct trace_array *tr) tr->gops = gops; gops->private = tr; + +fgraph_init_ops (&gops->ops, ops); +return 0 ; }
@@ -309,10 +312,11 @@ void free_fgraph_ops(struct trace_array *tr)kfree (tr->gops); } -__init void init_array_fgraph_ops (struct trace_array *tr) +__init void init_array_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops) { tr->gops = &funcgraph_ops; funcgraph_ops.private = tr; +fgraph_init_ops (&tr->gops->ops, ops); }static int graph_trace_init (struct trace_array *tr)
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 13/27 function_graph: Add pid tracing back to function graph tracer
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Now that the function_graph has a main callback that handles the function graph subops tracing, it no longer honors the pid filtering of ftrace. Add back this logic in the function_graph code to update the gops callback for the entry function to test if it should trace the current task or not.
--- include/linux/ftrace.h | 2 ++ kernel/trace/fgraph.c | 40 ++++++++++++++++++++++++++++++++++ kernel/trace/ftrace.c | 5 +++-- kernel/trace/ftrace_internal.h | 2 ++ 4 files changed, 47 insertions(+), 2 deletions(-)
include/linux/ftrace.h +2 -0
@@ -1040,6 +1040,7 @@ typedef int (*trace_func_graph_ent_t)(struct ftrace_graph_ent *, struct fgraph_ops *); /* entry */extern int ftrace_graph_entry_stub (struct ftrace_graph_ent *trace,struct fgraph_ops *gops); +bool ftrace_pids_enabled (struct ftrace_ops *ops);#ifdef CONFIG_FUNCTION_GRAPH_TRACER
@@ -1048,6 +1049,7 @@ struct fgraph_ops {trace_func_graph_ret_t retfunc;struct ftrace_ops ops;/* for the hash lists */ void *private; +trace_func_graph_ent_t saved_func;int idx; };
kernel/trace/fgraph.c +40 -0
@@ -854,6 +854,41 @@ void ftrace_graph_exit_task(struct task_struct *t)kfree (ret_stack); } +static int fgraph_pid_func (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) +{ +struct trace_array *tr = gops->ops.private; +int pid; + +if (tr) { + pid =this_cpu_read (tr->array_buffer.data->ftrace_ignore_pid); +if (pid ==FTRACE_PID_IGNORE ) +return 0 ; +if (pid !=FTRACE_PID_TRACE && + pid != current->pid) +return 0 ; + } + +return gops->saved_func (trace, gops); +} + +void fgraph_update_pid_func (void ) +{ +struct fgraph_ops *gops; +struct ftrace_ops *op; + +if (!(graph_ops.flags &FTRACE_OPS_FL_INITIALIZED )) +return ; + +list_for_each_entry (op, &graph_ops.subop_list, list) { +if (op->flags &FTRACE_OPS_FL_PID ) { + gops = container_of(op, struct fgraph_ops, ops); + gops->entryfunc =ftrace_pids_enabled (op) ? + fgraph_pid_func : gops->saved_func; + } + } +} +/* Allocate a return stack for each task */ static int start_graph_tracing (void ) {
@@ -931,11 +966,15 @@ int register_ftrace_graph(struct fgraph_ops *gops) command =FTRACE_START_FUNC_RET ; } +/* Always save the function, and reset at unregistering */ + gops->saved_func = gops->entryfunc; + ret =ftrace_startup_subops (&graph_ops, &gops->ops, command); error:if (ret) { fgraph_array[i] = &fgraph_stub; ftrace_graph_active--; + gops->saved_func =NULL ; } out:mutex_unlock (&ftrace_lock);
@@ -979,5 +1018,6 @@ void unregister_ftrace_graph(struct fgraph_ops *gops)unregister_trace_sched_switch (ftrace_graph_probe_sched_switch,NULL ); } out: + gops->saved_func =NULL ;mutex_unlock (&ftrace_lock); }
kernel/trace/ftrace.c +3 -2
@@ -100,7 +100,7 @@ struct ftrace_ops *function_trace_op __read_mostly = &ftrace_list_end;/* What to set function_trace_op to */ static struct ftrace_ops *set_function_trace_op; -static bool ftrace_pids_enabled (struct ftrace_ops *ops) +bool ftrace_pids_enabled (struct ftrace_ops *ops) {struct trace_array *tr;
@@ -402,10 +402,11 @@ static void ftrace_update_pid_func(void)if (op->flags &FTRACE_OPS_FL_PID ) { op->func =ftrace_pids_enabled (op) ? ftrace_pid_func : op->saved_func; -ftrace_update_trampoline (op); } }while_for_each_ftrace_op (op); +fgraph_update_pid_func (); +update_ftrace_function (); }
kernel/trace/ftrace_internal.h +2 -0
@@ -43,8 +43,10 @@ ftrace_ops_test(struct ftrace_ops *ops, unsigned long ip, void *regs)#ifdef CONFIG_FUNCTION_GRAPH_TRACER extern int ftrace_graph_active; +extern void fgraph_update_pid_func (void );#else /* !CONFIG_FUNCTION_GRAPH_TRACER */ # define ftrace_graph_active 0 +static inline void fgraph_update_pid_func (void ) {}#endif /* CONFIG_FUNCTION_GRAPH_TRACER */ #else /* !CONFIG_FUNCTION_TRACER */
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 14/27 function_graph: Use a simple LRU for fgraph_array index number
In reply to Steven RostedtFrom: "Masami Hiramatsu (Google)" <[email protected]>
Since the fgraph_array index is used for the bitmap on the shadow stack, it may leave some entries after a function_graph instance is removed. Thus if another instance reuses the fgraph_array index soon after releasing it, the fgraph may confuse to call the newer callback for the entries which are pushed by the older instance.
To avoid reusing the fgraph_array index soon after releasing, introduce a simple LRU table for managing the index number. This will reduce the possibility of this confusion.
--- kernel/trace/fgraph.c | 71 ++++++++++++++++++++++++++++++------------- 1 file changed, 50 insertions(+), 21 deletions(-)
kernel/trace/fgraph.c +50 -21
@@ -124,10 +124,48 @@ enum {DEFINE_STATIC_KEY_FALSE (kill_ftrace_graph);int ftrace_graph_active; -static int fgraph_array_cnt; -static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE ]; +/* LRU index table for fgraph_array */ +static int fgraph_lru_table[FGRAPH_ARRAY_SIZE ]; +static int fgraph_lru_next; +static int fgraph_lru_last; + +/* Initialize fgraph_lru_table with unused index */ +static void fgraph_lru_init (void ) +{ +int i; + +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) + fgraph_lru_table[i] = i; +} + +/* Release the used index to the LRU table */ +static int fgraph_lru_release_index (int idx) +{ +if (idx <0 || idx >=FGRAPH_ARRAY_SIZE || +WARN_ON_ONCE (fgraph_lru_table[fgraph_lru_last] !=-1 )) +return -1 ; + + fgraph_lru_table[fgraph_lru_last] = idx; + fgraph_lru_last = (fgraph_lru_last +1 ) %FGRAPH_ARRAY_SIZE ; +return 0 ; +} + +/* Allocate a new index from LRU table */ +static int fgraph_lru_alloc_index (void ) +{ +int idx = fgraph_lru_table[fgraph_lru_next]; + +/* No id is available */ +if (idx ==-1 ) +return -1 ; + + fgraph_lru_table[fgraph_lru_next] =-1 ; + fgraph_lru_next = (fgraph_lru_next +1 ) %FGRAPH_ARRAY_SIZE ; +return idx; +} +/* Get the FRAME_OFFSET from the word from the @offset on ret_stack */ static inline int get_frame_offset (struct task_struct *t,int offset) {
@@ -374,7 +412,7 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (offset <0 ) goto out; -for (i =0 ; i < fgraph_array_cnt; i++) { +for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) {struct fgraph_ops *gops = fgraph_array[i];if (gops == &fgraph_stub)
@@ -925,7 +963,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) {int command =0 ;int ret =0 ; -int i; +int i =-1 ;mutex_lock (&ftrace_lock);
@@ -933,21 +971,16 @@ int register_ftrace_graph(struct fgraph_ops *gops)/* The array must always have real data on it */ for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) fgraph_array[i] = &fgraph_stub; +fgraph_lru_init (); } -/* Look for an available spot */ -for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { -if (fgraph_array[i] == &fgraph_stub) -break ; - } -if (i >=FGRAPH_ARRAY_SIZE ) { + i =fgraph_lru_alloc_index (); +if (i <0 ||WARN_ON_ONCE (fgraph_array[i] != &fgraph_stub)) { ret = -ENOSPC ; goto out; } fgraph_array[i] = gops; -if (i +1 > fgraph_array_cnt) - fgraph_array_cnt = i +1 ; gops->idx = i; ftrace_graph_active++;
@@ -975,6 +1008,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) fgraph_array[i] = &fgraph_stub; ftrace_graph_active--; gops->saved_func =NULL ; +fgraph_lru_release_index (i); } out:mutex_unlock (&ftrace_lock);
@@ -984,25 +1018,20 @@ int register_ftrace_graph(struct fgraph_ops *gops)void unregister_ftrace_graph (struct fgraph_ops *gops) {int command =0 ; -int i;mutex_lock (&ftrace_lock);if (unlikely (!ftrace_graph_active)) goto out; -if (unlikely (gops->idx <0 || gops->idx >= fgraph_array_cnt)) +if (unlikely (gops->idx <0 || gops->idx >=FGRAPH_ARRAY_SIZE || + fgraph_array[gops->idx] != gops)) goto out; -WARN_ON_ONCE (fgraph_array[gops->idx] != gops); +if (fgraph_lru_release_index (gops->idx) <0 ) + goto out; fgraph_array[gops->idx] = &fgraph_stub; -if (gops->idx +1 == fgraph_array_cnt) { - i = gops->idx; -while (i >=0 && fgraph_array[i] == &fgraph_stub) - i--; - fgraph_array_cnt = i +1 ; - } ftrace_graph_active--;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 15/27 function_graph: Add "task variables" per task for fgraph_ops
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Add a "task variables" array on the tasks shadow ret_stack that is the size of longs for each possible registered fgraph_ops. That's a total of 16, taking up 8 * 16 = 128 bytes (out of a page size 4k).
This will allow for fgraph_ops to do specific features on a per task basis having a way to maintain state for each task.
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 1 + kernel/trace/fgraph.c | 74 +++++++++++++++++++++++++++++++++++++++++- 2 files changed, 74 insertions(+), 1 deletion(-)
include/linux/ftrace.h +1 -0
@@ -1089,6 +1089,7 @@ ftrace_graph_get_ret_stack(struct task_struct *task, int skip);unsigned long ftrace_graph_ret_addr (struct task_struct *task,int *idx,unsigned long ret,unsigned long *retp); +unsigned long *fgraph_get_task_var (struct fgraph_ops *gops);/* * Sometimes we don't want to trace a function with the function
kernel/trace/fgraph.c +73 -1
120 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 16/27 function_graph: Move set_graph_function tests to shadow stack global var
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
The use of the task->trace_recursion for the logic used for the set_graph_function was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
--- include/linux/trace_recursion.h | 5 +---- kernel/trace/trace.h | 32 ++++++++++++++++++---------- kernel/trace/trace_functions_graph.c | 6 +++--- kernel/trace/trace_irqsoff.c | 4 ++-- kernel/trace/trace_sched_wakeup.c | 4 ++-- 5 files changed, 29 insertions(+), 22 deletions(-)
include/linux/trace_recursion.h +1 -4
@@ -44,9 +44,6 @@ enum {*/ TRACE_IRQ_BIT, -/* Set if the function is in the set_graph_function file */ - TRACE_GRAPH_BIT, -/* * In the very unlikely case that an interrupt came in * at a start of graph tracing, and we want to trace
@@ -60,7 +57,7 @@ enum {* that preempted a softirq start of a function that * preempted normal context!!!! Luckily, it can't be * greater than 3, so the next two bits are a mask -* of what the depth is when we set TRACE_GRAPH_BIT +* of what the depth is when we set TRACE_GRAPH_FL */ TRACE_GRAPH_DEPTH_START_BIT,
kernel/trace/trace.h +21 -11
@@ -898,11 +898,16 @@ extern void init_array_fgraph_ops(struct trace_array *tr, struct ftrace_ops *opsextern int allocate_fgraph_ops (struct trace_array *tr,struct ftrace_ops *ops);extern void free_fgraph_ops (struct trace_array *tr); +enum { +TRACE_GRAPH_FL =1 , +}; +#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash; -static inline int ftrace_graph_addr (struct ftrace_graph_ent *trace) +static inline int +ftrace_graph_addr (unsigned long *task_var,struct ftrace_graph_ent *trace) {unsigned long addr = trace->func;int ret =0 ;
@@ -924,12 +929,11 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace) }if (ftrace_lookup_ip (hash, addr)) { -/* * This needs to be cleared on the return functions * when the depth is zero. */ -trace_recursion_set (TRACE_GRAPH_BIT ); + *task_var |=TRACE_GRAPH_FL ;trace_recursion_set_depth (trace->depth);/*
@@ -949,11 +953,14 @@ static inline int ftrace_graph_addr(struct ftrace_graph_ent *trace)return ret; } -static inline void ftrace_graph_addr_finish (struct ftrace_graph_ret *trace) +static inline void +ftrace_graph_addr_finish (struct fgraph_ops *gops,struct ftrace_graph_ret *trace) { -if (trace_recursion_test (TRACE_GRAPH_BIT ) && +unsigned long *task_var =fgraph_get_task_var (gops); + +if ((*task_var &TRACE_GRAPH_FL ) && trace->depth ==trace_recursion_depth ()) -trace_recursion_clear (TRACE_GRAPH_BIT ); + *task_var &= ~TRACE_GRAPH_FL ; }static inline int ftrace_graph_notrace_addr (unsigned long addr)
@@ -979,7 +986,7 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) return ret; }#else -static inline int ftrace_graph_addr (struct ftrace_graph_ent *trace) +static inline int ftrace_graph_addr (unsigned long *task_var,struct ftrace_graph_ent *trace) {return 1 ; }
@@ -988,17 +995,20 @@ static inline int ftrace_graph_notrace_addr(unsigned long addr) {return 0 ; } -static inline void ftrace_graph_addr_finish (struct ftrace_graph_ret *trace) +static inline void ftrace_graph_addr_finish (struct fgraph_ops *gops,struct ftrace_graph_ret *trace) { }#endif /* CONFIG_DYNAMIC_FTRACE */ extern unsigned int fgraph_max_depth; -static inline bool ftrace_graph_ignore_func (struct ftrace_graph_ent *trace) +static inline bool +ftrace_graph_ignore_func (struct fgraph_ops *gops,struct ftrace_graph_ent *trace) { +unsigned long *task_var =fgraph_get_task_var (gops); +/* trace it when it is-nested-in or is a function enabled. */ -return !(trace_recursion_test (TRACE_GRAPH_BIT ) || -ftrace_graph_addr (trace)) || +return !((*task_var &TRACE_GRAPH_FL ) || +ftrace_graph_addr (task_var, trace)) || (trace->depth <0 ) || (fgraph_max_depth && trace->depth >= fgraph_max_depth); }
kernel/trace/trace_functions_graph.c +3 -3
@@ -160,7 +160,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,if (!ftrace_trace_task (tr))return 0 ; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;if (ftrace_graph_ignore_irqs ())
@@ -247,7 +247,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace,long disabled;int cpu; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) {trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT );
@@ -269,7 +269,7 @@ void trace_graph_return(struct ftrace_graph_ret *trace,static void trace_graph_thresh_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) {trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT );
kernel/trace/trace_irqsoff.c +2 -2
@@ -184,7 +184,7 @@ static int irqsoff_graph_entry(struct ftrace_graph_ent *trace,unsigned int trace_ctx;int ret; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;/* * Do not trace a function if it's filtered by set_graph_notrace.
@@ -214,7 +214,7 @@ static void irqsoff_graph_return(struct ftrace_graph_ret *trace,unsigned long flags;unsigned int trace_ctx; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (!func_prolog_dec (tr, &data, &flags))return ;
kernel/trace/trace_sched_wakeup.c +2 -2
@@ -120,7 +120,7 @@ static int wakeup_graph_entry(struct ftrace_graph_ent *trace,unsigned int trace_ctx;int ret =0 ; -if (ftrace_graph_ignore_func (trace)) +if (ftrace_graph_ignore_func (gops, trace))return 0 ;/* * Do not trace a function if it's filtered by set_graph_notrace.
@@ -149,7 +149,7 @@ static void wakeup_graph_return(struct ftrace_graph_ret *trace,struct trace_array_cpu *data;unsigned int trace_ctx; -ftrace_graph_addr_finish (trace); +ftrace_graph_addr_finish (gops, trace);if (!func_prolog_preempt_disable (tr, &data, &trace_ctx))return ;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 17/27 function_graph: Move graph depth stored data to shadow stack global var
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
The use of the task->trace_recursion for the logic used for the function graph depth was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
--- include/linux/trace_recursion.h | 29 ---------------------------- kernel/trace/trace.h | 34 +++++++++++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 31 deletions(-)
include/linux/trace_recursion.h +0 -29
@@ -44,25 +44,6 @@ enum {*/ TRACE_IRQ_BIT, -/* -* In the very unlikely case that an interrupt came in -* at a start of graph tracing, and we want to trace -* the function in that interrupt, the depth can be greater -* than zero, because of the preempted start of a previous -* trace. In an even more unlikely case, depth could be 2 -* if a softirq interrupted the start of graph tracing, -* followed by an interrupt preempting a start of graph -* tracing in the softirq, and depth can even be 3 -* if an NMI came in at the start of an interrupt function -* that preempted a softirq start of a function that -* preempted normal context!!!! Luckily, it can't be -* greater than 3, so the next two bits are a mask -* of what the depth is when we set TRACE_GRAPH_FL -*/ - -TRACE_GRAPH_DEPTH_START_BIT , -TRACE_GRAPH_DEPTH_END_BIT , -/* * To implement set_graph_notrace, if this bit is set, we ignore * function graph tracing of called functions, until the return
@@ -78,16 +59,6 @@ enum {#define trace_recursion_clear (bit) do { (current)->trace_recursion &= ~(1<<(bit)); } while (0)#define trace_recursion_test (bit) ((current)->trace_recursion & (1<<(bit))) -#define trace_recursion_depth () \ - (((current)->trace_recursion >> TRACE_GRAPH_DEPTH_START_BIT) & 3) -#define trace_recursion_set_depth (depth) \ - do { \ - current->trace_recursion &= \ - ~(3 << TRACE_GRAPH_DEPTH_START_BIT); \ - current->trace_recursion |= \ - ((depth) & 3) << TRACE_GRAPH_DEPTH_START_BIT; \ - } while (0) -#define TRACE_CONTEXT_BITS 4#define TRACE_FTRACE_START TRACE_FTRACE_BIT
kernel/trace/trace.h +32 -2
@@ -900,8 +900,38 @@ extern void free_fgraph_ops(struct trace_array *tr);enum {TRACE_GRAPH_FL =1 , + +/* +* In the very unlikely case that an interrupt came in +* at a start of graph tracing, and we want to trace +* the function in that interrupt, the depth can be greater +* than zero, because of the preempted start of a previous +* trace. In an even more unlikely case, depth could be 2 +* if a softirq interrupted the start of graph tracing, +* followed by an interrupt preempting a start of graph +* tracing in the softirq, and depth can even be 3 +* if an NMI came in at the start of an interrupt function +* that preempted a softirq start of a function that +* preempted normal context!!!! Luckily, it can't be +* greater than 3, so the next two bits are a mask +* of what the depth is when we set TRACE_GRAPH_FL +*/ + +TRACE_GRAPH_DEPTH_START_BIT , +TRACE_GRAPH_DEPTH_END_BIT , }; +static inline unsigned long ftrace_graph_depth (unsigned long *task_var) +{ +return (*task_var >>TRACE_GRAPH_DEPTH_START_BIT ) &3 ; +} + +static inline void ftrace_graph_set_depth (unsigned long *task_var,int depth) +{ + *task_var &= ~(3 <<TRACE_GRAPH_DEPTH_START_BIT ); + *task_var |= (depth &3 ) <<TRACE_GRAPH_DEPTH_START_BIT ; +} +#ifdef CONFIG_DYNAMIC_FTRACE extern struct ftrace_hash __rcu *ftrace_graph_hash;extern struct ftrace_hash __rcu *ftrace_graph_notrace_hash;
@@ -934,7 +964,7 @@ ftrace_graph_addr(unsigned long *task_var, struct ftrace_graph_ent *trace)* when the depth is zero. */ *task_var |=TRACE_GRAPH_FL ; -trace_recursion_set_depth (trace->depth); +ftrace_graph_set_depth (task_var, trace->depth);/* * If no irqs are to be traced, but a set_graph_function
@@ -959,7 +989,7 @@ ftrace_graph_addr_finish(struct fgraph_ops *gops, struct ftrace_graph_ret *traceunsigned long *task_var =fgraph_get_task_var (gops);if ((*task_var &TRACE_GRAPH_FL ) && - trace->depth ==trace_recursion_depth ()) + trace->depth ==ftrace_graph_depth (task_var)) *task_var &= ~TRACE_GRAPH_FL ; }
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 18/27 function_graph: Move graph notrace bit to shadow stack global var
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
The use of the task->trace_recursion for the logic used for the function graph no-trace was a bit of an abuse of that variable. Now that there exists global vars that are per stack for registered graph traces, use that instead.
--- include/linux/trace_recursion.h | 7 ------- kernel/trace/trace.h | 9 +++++++++ kernel/trace/trace_functions_graph.c | 10 ++++++---- 3 files changed, 15 insertions(+), 11 deletions(-)
include/linux/trace_recursion.h +0 -7
@@ -44,13 +44,6 @@ enum {*/ TRACE_IRQ_BIT , -/* -* To implement set_graph_notrace, if this bit is set, we ignore -* function graph tracing of called functions, until the return -* function is called to clear it. -*/ -TRACE_GRAPH_NOTRACE_BIT , -/* Used to prevent recursion recording from recursing. */ TRACE_RECORD_RECURSION_BIT, };
kernel/trace/trace.h +9 -0
@@ -919,8 +919,17 @@ enum {TRACE_GRAPH_DEPTH_START_BIT ,TRACE_GRAPH_DEPTH_END_BIT , + +/* +* To implement set_graph_notrace, if this bit is set, we ignore +* function graph tracing of called functions, until the return +* function is called to clear it. +*/ + TRACE_GRAPH_NOTRACE_BIT, }; +#define TRACE_GRAPH_NOTRACE (1 << TRACE_GRAPH_NOTRACE_BIT) +static inline unsigned long ftrace_graph_depth (unsigned long *task_var) {return (*task_var >>TRACE_GRAPH_DEPTH_START_BIT ) &3 ;
kernel/trace/trace_functions_graph.c +6 -4
@@ -130,6 +130,7 @@ static inline int ftrace_graph_ignore_irqs(void)int trace_graph_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops) { +unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;struct trace_array_cpu *data;unsigned long flags;
@@ -138,7 +139,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,int ret;int cpu; -if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) +if (*task_var &TRACE_GRAPH_NOTRACE )return 0 ;/*
@@ -149,7 +150,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,* returning from the function. */ if (ftrace_graph_notrace_addr (trace->func)) { -trace_recursion_set (TRACE_GRAPH_NOTRACE_BIT ); + *task_var |=TRACE_GRAPH_NOTRACE_BIT ;/* * Need to return 1 to have the return called * that will clear the NOTRACE bit.
@@ -240,6 +241,7 @@ void __trace_graph_return(struct trace_array *tr,void trace_graph_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { +unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;struct trace_array_cpu *data;unsigned long flags;
@@ -249,8 +251,8 @@ void trace_graph_return(struct ftrace_graph_ret *trace,ftrace_graph_addr_finish (gops, trace); -if (trace_recursion_test (TRACE_GRAPH_NOTRACE_BIT )) { -trace_recursion_clear (TRACE_GRAPH_NOTRACE_BIT ); +if (*task_var &TRACE_GRAPH_NOTRACE ) { + *task_var &= ~TRACE_GRAPH_NOTRACE ;return ; }
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 19/27 function_graph: Implement fgraph_reserve_data() and fgraph_retrieve_data()
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Added functions that can be called by a fgraph_ops entryfunc and retfunc to store state between the entry of the function being traced to the exit of the same function. The fgraph_ops entryfunc() may call fgraph_reserve_data() to store up to 32 words onto the task's shadow ret_stack and this then can be retrieved by fgraph_retrieve_data() called by the corresponding retfunc().
Co-developed with Masami Hiramatsu:
--- include/linux/ftrace.h | 3 + kernel/trace/fgraph.c | 191 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 186 insertions(+), 8 deletions(-)
include/linux/ftrace.h +3 -0
@@ -1053,6 +1053,9 @@ struct fgraph_ops {int idx; }; +void *fgraph_reserve_data (int idx,int size_bytes); +void *fgraph_retrieve_data (int idx,int *size_bytes); +/* * Stack of return addresses for functions * of a thread.
kernel/trace/fgraph.c +183 -8
276 lines not on this page · raw message
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 20/27 function_graph: Add selftest for passing local variables
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
Add boot up selftest that passes variables from a function entry to a function exit, and make sure that they do get passed around.
Co-developed with Masami Hiramatsu:
--- kernel/trace/trace_selftest.c | 169 ++++++++++++++++++++++++++++++++++ 1 file changed, 169 insertions(+)
kernel/trace/trace_selftest.c +169 -0
@@ -756,6 +756,173 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)#ifdef CONFIG_FUNCTION_GRAPH_TRACER +#ifdef CONFIG_DYNAMIC_FTRACE + +#define BYTE_NUMBER 123 +#define SHORT_NUMBER 12345 +#define WORD_NUMBER 1234567890 +#define LONG_NUMBER 1234567890123456789LL + +static int fgraph_store_size__initdata ; +static const char *fgraph_store_type_name__initdata ; +static char *fgraph_error_str__initdata ; +static char fgraph_error_str_buf[128 ]__initdata ; + +static __init int store_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops) +{ +const char *type = fgraph_store_type_name; +int size = fgraph_store_size; +void *p; + + p =fgraph_reserve_data (gops->idx, size); +if (!p) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to reserve %s\n" , type); + fgraph_error_str = fgraph_error_str_buf; +return 0 ; + } + +switch (fgraph_store_size) { +case 1 : + *(char *)p =BYTE_NUMBER ; +break ; +case 2 : + *(short *)p =SHORT_NUMBER ; +break ; +case 4 : + *(int *)p =WORD_NUMBER ; +break ; +case 8 : + *(long long *)p =LONG_NUMBER ; +break ; + } + +return 1 ; +} + +static __init void store_return (struct ftrace_graph_ret *trace, +struct fgraph_ops *gops) +{ +const char *type = fgraph_store_type_name; +long long expect =0 ; +long long found =-1 ; +int size; +char *p; + + p =fgraph_retrieve_data (gops->idx, &size); +if (!p) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to retrieve %s\n" , type); + fgraph_error_str = fgraph_error_str_buf; +return ; + } +if (fgraph_store_size > size) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Retrieved size %d is smaller than expected %d\n" , + size, (int )fgraph_store_size); + fgraph_error_str = fgraph_error_str_buf; +return ; + } + +switch (fgraph_store_size) { +case 1 : + expect =BYTE_NUMBER ; + found = *(char *)p; +break ; +case 2 : + expect =SHORT_NUMBER ; + found = *(short *)p; +break ; +case 4 : + expect =WORD_NUMBER ; + found = *(int *)p; +break ; +case 8 : + expect =LONG_NUMBER ; + found = *(long long *)p; +break ; + } + +if (found != expect) { +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"%s returned not %lld but %lld\n" , type, expect, found); + fgraph_error_str = fgraph_error_str_buf; +return ; + } + fgraph_error_str =NULL ; +} + +static struct fgraph_ops store_bytes__initdata = { + .entryfunc = store_entry, + .retfunc = store_return, +}; + +static int __init test_graph_storage_type (const char *name,int size) +{ +char *func_name; +int len; +int ret; + + fgraph_store_type_name = name; + fgraph_store_size = size; + +snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +"Failed to execute storage %s\n" , name); + fgraph_error_str = fgraph_error_str_buf; + +pr_cont ("PASSED\n" ); +pr_info ("Testing fgraph storage of %d byte%s: " , size, size >1 ?"s" :"" ); + + func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); + len =strlen (func_name); + + ret =ftrace_set_filter (&store_bytes.ops, func_name, len,1 ); +if (ret && ret != -ENODEV ) { +pr_cont ("*Could not set filter* " ); +return -1 ; + } + + ret =register_ftrace_graph (&store_bytes); +if (ret) { +pr_warn ("Failed to init store_bytes fgraph tracing\n" ); +return -1 ; + } + +DYN_FTRACE_TEST_NAME (); + +unregister_ftrace_graph (&store_bytes); + +if (fgraph_error_str) { +pr_cont ("*** %s ***" , fgraph_error_str); +return -1 ; + } + +return 0 ; +} +/* Test the storage passed across function_graph entry and return */ +static __init int test_graph_storage (void ) +{ +int ret; + + ret =test_graph_storage_type ("byte" ,1 ); +if (ret) +return ret; + ret =test_graph_storage_type ("short" ,2 ); +if (ret) +return ret; + ret =test_graph_storage_type ("word" ,4 ); +if (ret) +return ret; + ret =test_graph_storage_type ("long long" ,8 ); +if (ret) +return ret; +return 0 ; +} +#else +static inline int test_graph_storage (void ) {return 0 ; } +#endif /* CONFIG_DYNAMIC_FTRACE */ +/* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000
@@ -913,6 +1080,8 @@ trace_selftest_startup_function_graph(struct tracer *trace,ftrace_set_global_filter (NULL ,0 ,1 );#endif + ret =test_graph_storage (); +/* Don't test dynamic tracing, the function tracer already did */ out:/* Stop it if we failed */
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 21/27 ftrace: Add multiple fgraph storage selftest
In reply to Steven RostedtFrom: "Masami Hiramatsu (Google)" <[email protected]>
Add a selftest for multiple function graph tracer with storage on a same function. In this case, the shadow stack entry will be shared among those fgraph with different data storage. So this will ensure the fgraph will not mixed those storage data.
--- kernel/trace/trace_selftest.c | 171 +++++++++++++++++++++++++--------- 1 file changed, 126 insertions(+), 45 deletions(-)
kernel/trace/trace_selftest.c +126 -45
@@ -762,28 +762,32 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)#define SHORT_NUMBER 12345#define WORD_NUMBER 1234567890#define LONG_NUMBER 1234567890123456789LL - -static int fgraph_store_size__initdata ; -static const char *fgraph_store_type_name__initdata ; -static char *fgraph_error_str__initdata ; -static char fgraph_error_str_buf[128 ]__initdata ; +#define ERRSTR_BUFLEN 128 + +struct fgraph_fixture { +struct fgraph_ops gops; +int store_size; +const char *store_type_name; +char error_str_buf[ERRSTR_BUFLEN ]; +char *error_str; +};static __init int store_entry (struct ftrace_graph_ent *trace,struct fgraph_ops *gops) { -const char *type = fgraph_store_type_name; -int size = fgraph_store_size; + struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops); +const char *type = fixture->store_type_name; +int size = fixture->store_size;void *p; p =fgraph_reserve_data (gops->idx, size);if (!p) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Failed to reserve %s\n" , type); - fgraph_error_str = fgraph_error_str_buf;return 0 ; } -switch (fgraph_store_size) { +switch (size) {case 1 : *(char *)p =BYTE_NUMBER ;break ;
@@ -804,7 +808,8 @@ static __init int store_entry(struct ftrace_graph_ent *trace,static __init void store_return (struct ftrace_graph_ret *trace,struct fgraph_ops *gops) { -const char *type = fgraph_store_type_name; + struct fgraph_fixture *fixture = container_of(gops, struct fgraph_fixture, gops); +const char *type = fixture->store_type_name;long long expect =0 ;long long found =-1 ;int size;
@@ -812,20 +817,18 @@ static __init void store_return(struct ftrace_graph_ret *trace, p =fgraph_retrieve_data (gops->idx, &size);if (!p) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Failed to retrieve %s\n" , type); - fgraph_error_str = fgraph_error_str_buf;return ; } -if (fgraph_store_size > size) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +if (fixture->store_size > size) { +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"Retrieved size %d is smaller than expected %d\n" , - size, (int )fgraph_store_size); - fgraph_error_str = fgraph_error_str_buf; + size, (int )fixture->store_size);return ; } -switch (fgraph_store_size) { +switch (fixture->store_size) {case 1 : expect =BYTE_NUMBER ; found = *(char *)p;
@@ -845,45 +848,44 @@ static __init void store_return(struct ftrace_graph_ret *trace, }if (found != expect) { -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN ,"%s returned not %lld but %lld\n" , type, expect, found); - fgraph_error_str = fgraph_error_str_buf;return ; } - fgraph_error_str =NULL ; + fixture->error_str =NULL ; } -static struct fgraph_ops store_bytes__initdata = { - .entryfunc = store_entry, - .retfunc = store_return, -}; - -static int __init test_graph_storage_type (const char *name,int size) +static int __init init_fgraph_fixture (struct fgraph_fixture *fixture) {char *func_name;int len; -int ret; - fgraph_store_type_name = name; - fgraph_store_size = size; +snprintf (fixture->error_str_buf,ERRSTR_BUFLEN , +"Failed to execute storage %s\n" , fixture->store_type_name); + fixture->error_str = fixture->error_str_buf; -snprintf (fgraph_error_str_buf,sizeof (fgraph_error_str_buf), -"Failed to execute storage %s\n" , name); - fgraph_error_str = fgraph_error_str_buf; + func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); + len =strlen (func_name); + +return ftrace_set_filter (&fixture->gops.ops, func_name, len,1 ); +} + +/* Test fgraph storage for each size */ +static int __init test_graph_storage_single (struct fgraph_fixture *fixture) +{ +int size = fixture->store_size; +int ret;pr_cont ("PASSED\n" );pr_info ("Testing fgraph storage of %d byte%s: " , size, size >1 ?"s" :"" ); - func_name ="*" __stringify(DYN_FTRACE_TEST_NAME ); - len =strlen (func_name); - - ret =ftrace_set_filter (&store_bytes.ops, func_name, len,1 ); + ret =init_fgraph_fixture (fixture);if (ret && ret != -ENODEV ) {pr_cont ("*Could not set filter* " );return -1 ; } - ret =register_ftrace_graph (&store_bytes); + ret =register_ftrace_graph (&fixture->gops);if (ret) {pr_warn ("Failed to init store_bytes fgraph tracing\n" );return -1 ;
@@ -891,30 +893,109 @@ static int __init test_graph_storage_type(const char *name, int size)DYN_FTRACE_TEST_NAME (); -unregister_ftrace_graph (&store_bytes); +unregister_ftrace_graph (&fixture->gops); -if (fgraph_error_str) { -pr_cont ("*** %s ***" , fgraph_error_str); +if (fixture->error_str) { +pr_cont ("*** %s ***" , fixture->error_str);return -1 ; }return 0 ; } + +static struct fgraph_fixture store_bytes[4 ]__initdata = { + [0 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =1 , + .store_type_name ="byte" , + }, + [1 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =2 , + .store_type_name ="short" , + }, + [2 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =4 , + .store_type_name ="word" , + }, + [3 ] = { + .gops = { + .entryfunc = store_entry, + .retfunc = store_return, + }, + .store_size =8 , + .store_type_name ="long long" , + }, +}; + +static __init int test_graph_storage_multi (void ) +{ +struct fgraph_fixture *fixture; +bool printed = false; +int i, ret; + +pr_cont ("PASSED\n" ); +pr_info ("Testing multiple fgraph storage on a function: " ); + +for (i =0 ; i <ARRAY_SIZE (store_bytes); i++) { + fixture = &store_bytes[i]; + ret =init_fgraph_fixture (fixture); +if (ret && ret != -ENODEV ) { +pr_cont ("*Could not set filter* " ); + printed = true; + goto out; + } + + ret =register_ftrace_graph (&fixture->gops); +if (ret) { +pr_warn ("Failed to init store_bytes fgraph tracing\n" ); + printed = true; + goto out; + } + } + +DYN_FTRACE_TEST_NAME (); +out: +while (--i >=0 ) { + fixture = &store_bytes[i]; +unregister_ftrace_graph (&fixture->gops); + +if (fixture->error_str && !printed) { +pr_cont ("*** %s ***" , fixture->error_str); + printed = true; + } + } +return printed ?-1 :0 ; +} +/* Test the storage passed across function_graph entry and return */ static __init int test_graph_storage (void ) {int ret; - ret =test_graph_storage_type ("byte" ,1 ); + ret =test_graph_storage_single (&store_bytes[0 ]); +if (ret) +return ret; + ret =test_graph_storage_single (&store_bytes[1 ]);if (ret)return ret; - ret =test_graph_storage_type ("short" ,2 ); + ret =test_graph_storage_single (&store_bytes[2 ]);if (ret)return ret; - ret =test_graph_storage_type ("word" ,4 ); + ret =test_graph_storage_single (&store_bytes[3 ]);if (ret)return ret; - ret =test_graph_storage_type ("long long" ,8 ); + ret =test_graph_storage_multi ();if (ret)return ret;return 0 ;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 22/27 function_graph: Use for_each_set_bit() in __ftrace_return_to_handler()
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Instead of iterating through the entire fgraph_array[] and seeing if one of the bitmap bits are set to know to call the array's retfunc() function, use for_each_set_bit() on the bitmap itself. This will only iterate for the number of set bits.
--- kernel/trace/fgraph.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
kernel/trace/fgraph.c +2 -3
@@ -783,11 +783,10 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs#endif bitmap =get_bitmap_bits (current, offset); -for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { + +for_each_set_bit (i, &bitmap,sizeof (bitmap) *BITS_PER_BYTE ) {struct fgraph_ops *gops = fgraph_array[i]; -if (!(bitmap &BIT (i))) -continue ;if (gops == &fgraph_stub)continue ;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 23/27 function_graph: Use bitmask to loop on fgraph entry
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Instead of looping through all the elements of fgraph_array[] to see if there's an gops attached to one and then calling its gops->func(). Create a fgraph_array_bitmask that sets bits when an index in the array is reserved (via the simple lru algorithm). Then only the bits set in this bitmask needs to be looked at where only elements in the array that have ops registered need to be looked at.
Note, we do not care about races. If a bit is set before the gops is assigned, it only wastes time looking at the element and ignoring it (as it did before this bitmask is added).
--- kernel/trace/fgraph.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
kernel/trace/fgraph.c +7 -1
@@ -172,6 +172,7 @@ DEFINE_STATIC_KEY_FALSE(kill_ftrace_graph);int ftrace_graph_active;static struct fgraph_ops *fgraph_array[FGRAPH_ARRAY_SIZE ]; +static unsigned long fgraph_array_bitmask;/* LRU index table for fgraph_array */ static int fgraph_lru_table[FGRAPH_ARRAY_SIZE ];
@@ -196,6 +197,8 @@ static int fgraph_lru_release_index(int idx) fgraph_lru_table[fgraph_lru_last] = idx; fgraph_lru_last = (fgraph_lru_last +1 ) %FGRAPH_ARRAY_SIZE ; + +clear_bit (idx, &fgraph_array_bitmask);return 0 ; }
@@ -210,6 +213,8 @@ static int fgraph_lru_alloc_index(void) fgraph_lru_table[fgraph_lru_next] =-1 ; fgraph_lru_next = (fgraph_lru_next +1 ) %FGRAPH_ARRAY_SIZE ; + +set_bit (idx, &fgraph_array_bitmask);return idx; }
@@ -631,7 +636,8 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (offset <0 ) goto out; -for (i =0 ; i <FGRAPH_ARRAY_SIZE ; i++) { +for_each_set_bit (i, &fgraph_array_bitmask, +sizeof (fgraph_array_bitmask) *BITS_PER_BYTE ) {struct fgraph_ops *gops = fgraph_array[i];int save_curr_ret_stack;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 24/27 function_graph: Use static_call and branch to optimize entry function
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
In most cases function graph is used by a single user. Instead of calling a loop to call function graph callbacks in this case, call the function entry callback directly.
Add a static_key that will be used to set the function graph logic to either do the loop (when more than one callback is registered) or to call the callback directly if there is only one registered callback.
--- kernel/trace/fgraph.c | 77 ++++++++++++++++++++++++++++++++++++------- 1 file changed, 66 insertions(+), 11 deletions(-)
kernel/trace/fgraph.c +66 -11
@@ -11,6 +11,7 @@#include <linux/jump_label.h> #include <linux/suspend.h> #include <linux/ftrace.h> +#include <linux/static_call.h> #include <linux/slab.h> #include <trace/events/sched.h>
@@ -511,6 +512,10 @@ static struct fgraph_ops fgraph_stub = { .retfunc = ftrace_graph_ret_stub, }; +static struct fgraph_ops *fgraph_direct_gops = &fgraph_stub; +DEFINE_STATIC_CALL (fgraph_func, ftrace_graph_entry_stub); +DEFINE_STATIC_KEY_TRUE (fgraph_do_direct); +/** * ftrace_graph_stop - set to permanently disable function graph tracing *
@@ -636,21 +641,34 @@ int function_graph_enter(unsigned long ret, unsigned long func,if (offset <0 ) goto out; -for_each_set_bit (i, &fgraph_array_bitmask, -sizeof (fgraph_array_bitmask) *BITS_PER_BYTE ) { -struct fgraph_ops *gops = fgraph_array[i]; -int save_curr_ret_stack; - -if (gops == &fgraph_stub) -continue ; +#ifdef CONFIG_HAVE_STATIC_CALL +if (static_branch_likely (&fgraph_do_direct)) { +int save_curr_ret_stack = current->curr_ret_stack; - save_curr_ret_stack = current->curr_ret_stack; -if (ftrace_ops_test (&gops->ops, func,NULL ) && - gops->entryfunc (&trace, gops)) - bitmap |=BIT (i); +if (static_call (fgraph_func)(&trace, fgraph_direct_gops)) + bitmap |=BIT (fgraph_direct_gops->idx);else /* Clear out any saved storage */ current->curr_ret_stack = save_curr_ret_stack; + } else +#endif + { +for_each_set_bit (i, &fgraph_array_bitmask, +sizeof (fgraph_array_bitmask) *BITS_PER_BYTE ) { +struct fgraph_ops *gops = fgraph_array[i]; +int save_curr_ret_stack; + +if (gops == &fgraph_stub) +continue ; + + save_curr_ret_stack = current->curr_ret_stack; +if (ftrace_ops_test (&gops->ops, func,NULL ) && + gops->entryfunc (&trace, gops)) + bitmap |=BIT (i); +else +/* Clear out any saved storage */ + current->curr_ret_stack = save_curr_ret_stack; + } }if (!bitmap)
@@ -1155,6 +1173,8 @@ void fgraph_update_pid_func(void) gops = container_of(op, struct fgraph_ops, ops); gops->entryfunc =ftrace_pids_enabled (op) ? fgraph_pid_func : gops->saved_func; +if (ftrace_graph_active ==1 ) +static_call_update (fgraph_func, gops->entryfunc); } } }
@@ -1209,6 +1229,32 @@ static void init_task_vars(int idx)read_unlock (&tasklist_lock); } +static void ftrace_graph_enable_direct (bool enable_branch) +{ +trace_func_graph_ent_t func =NULL ; +int i; + +for_each_set_bit (i, &fgraph_array_bitmask, +sizeof (fgraph_array_bitmask) *BITS_PER_BYTE ) { + func = fgraph_array[i]->entryfunc; + fgraph_direct_gops = fgraph_array[i]; + } +if (WARN_ON_ONCE (!func)) +return ; + +static_call_update (fgraph_func, func); +if (enable_branch) +static_branch_disable (&fgraph_do_direct); +} + +static void ftrace_graph_disable_direct (bool disable_branch) +{ +if (disable_branch) +static_branch_disable (&fgraph_do_direct); +static_call_update (fgraph_func, ftrace_graph_entry_stub); + fgraph_direct_gops = &fgraph_stub; +} +int register_ftrace_graph (struct fgraph_ops *gops) {int command =0 ;
@@ -1235,7 +1281,11 @@ int register_ftrace_graph(struct fgraph_ops *gops) ftrace_graph_active++; +if (ftrace_graph_active ==2 ) +ftrace_graph_disable_direct (true); +if (ftrace_graph_active ==1 ) { +ftrace_graph_enable_direct (false);register_pm_notifier (&ftrace_suspend_notifier); ret =start_graph_tracing ();if (ret)
@@ -1292,6 +1342,11 @@ void unregister_ftrace_graph(struct fgraph_ops *gops)ftrace_shutdown_subops (&graph_ops, &gops->ops, command); +if (ftrace_graph_active ==1 ) +ftrace_graph_enable_direct (true); +else if (!ftrace_graph_active) +ftrace_graph_disable_direct (false); +if (!ftrace_graph_active) { ftrace_graph_return = ftrace_stub_graph; ftrace_graph_entry = ftrace_graph_entry_stub;
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 25/27 function_graph: Use static_call and branch to optimize return function
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
In most cases function graph is used by a single user. Instead of calling a loop to call function graph callbacks in this case, call the function return callback directly.
Use the static_key that is set when the function graph tracer has less than 2 callbacks registered. It will do the direct call in that case, and will do the loop over all callers when there are 2 or more callbacks registered.
--- kernel/trace/fgraph.c | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-)
kernel/trace/fgraph.c +18 -5
@@ -514,6 +514,7 @@ static struct fgraph_ops fgraph_stub = {static struct fgraph_ops *fgraph_direct_gops = &fgraph_stub;DEFINE_STATIC_CALL (fgraph_func, ftrace_graph_entry_stub); +DEFINE_STATIC_CALL (fgraph_retfunc, ftrace_graph_ret_stub);DEFINE_STATIC_KEY_TRUE (fgraph_do_direct);/**
@@ -808,13 +809,21 @@ static unsigned long __ftrace_return_to_handler(struct fgraph_ret_regs *ret_regs bitmap =get_bitmap_bits (current, offset); -for_each_set_bit (i, &bitmap,sizeof (bitmap) *BITS_PER_BYTE ) { -struct fgraph_ops *gops = fgraph_array[i]; +#ifdef CONFIG_HAVE_STATIC_CALL +if (static_branch_likely (&fgraph_do_direct)) { +if (test_bit (fgraph_direct_gops->idx, &bitmap)) +static_call (fgraph_retfunc)(&trace, fgraph_direct_gops); + } else +#endif + { +for_each_set_bit (i, &bitmap,sizeof (bitmap) *BITS_PER_BYTE ) { +struct fgraph_ops *gops = fgraph_array[i]; -if (gops == &fgraph_stub) -continue ; +if (gops == &fgraph_stub) +continue ; - gops->retfunc (&trace, gops); + gops->retfunc (&trace, gops); + } }/*
@@ -1232,17 +1241,20 @@ static void init_task_vars(int idx)static void ftrace_graph_enable_direct (bool enable_branch) {trace_func_graph_ent_t func =NULL ; +trace_func_graph_ret_t retfunc =NULL ;int i;for_each_set_bit (i, &fgraph_array_bitmask,sizeof (fgraph_array_bitmask) *BITS_PER_BYTE ) { func = fgraph_array[i]->entryfunc; + retfunc = fgraph_array[i]->retfunc; fgraph_direct_gops = fgraph_array[i]; }if (WARN_ON_ONCE (!func))return ;static_call_update (fgraph_func, func); +static_call_update (fgraph_retfunc, retfunc);if (enable_branch)static_branch_disable (&fgraph_do_direct); }
@@ -1252,6 +1264,7 @@ static void ftrace_graph_disable_direct(bool disable_branch)if (disable_branch)static_branch_disable (&fgraph_do_direct);static_call_update (fgraph_func, ftrace_graph_entry_stub); +static_call_update (fgraph_retfunc, ftrace_graph_ret_stub); fgraph_direct_gops = &fgraph_stub; }
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 26/27 selftests/ftrace: Add function_graph tracer to func-filter-pid test
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The function tracer is tested to see if pid filtering works. Add a test to test function_graph tracer as well, but only if the function_graph tracer is enabled for the top level or instance.
--- .../ftrace/test.d/ftrace/func-filter-pid.tc | 27 +++++++++++++++---- 1 file changed, 22 insertions(+), 5 deletions(-)
tools/testing/selftests/ftrace/test.d/ftrace/func-filter-pid.tc +22 -5
@@ -14,6 +14,11 @@ if [ ! -f options/function-fork ]; then echo "no option for function-fork found. Option will not be tested." fi +if [ ! -f options/funcgraph-proc ]; then + do_funcgraph_proc=0 + echo "no option for function-fork found. Option will not be tested." +fi + read PID _ < /proc/self/stat if [ $do_function_fork -eq 1 ]; then
@@ -21,12 +26,18 @@ if [ $do_function_fork -eq 1 ]; then orig_value=`grep function-fork trace_options` fi +if [ $do_funcgraph_proc -eq 1 ]; then + orig_value2=`cat options/funcgraph-proc` +fi + do_reset() { - if [ $do_function_fork -eq 0 ]; then - return + if [ $do_function_fork -eq 1 ]; then + echo $orig_value > trace_options fi - echo $orig_value > trace_options + if [ $do_funcgraph_proc -eq 1 ]; then + echo $orig_value2 > options/funcgraph-proc + fi } fail() { # msg
@@ -36,13 +47,15 @@ fail() { # msg } do_test() { + TRACER=$1 + disable_tracing echo do_execve* > set_ftrace_filter echo $FUNCTION_FORK >> set_ftrace_filter echo $PID > set_ftrace_pid - echo function > current_tracer + echo $TRACER > current_tracer if [ $do_function_fork -eq 1 ]; then # don't allow children to be traced
@@ -82,7 +95,11 @@ do_test() { fi } -do_test +do_test function +if grep -s function_graph available_tracers; then + do_test function_graph +fi + do_reset exit 0
Signature · 1 line
2.43.0
Steven Rostedt
maintainerPATCH v3 27/27 selftests/ftrace: Add fgraph-multi.tc test
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Add a test that creates 3 instances and enables function_graph tracer in each as well as the top instance, where each will enable a filter (but one that traces all functions) and check that they are filtering properly.
--- .../ftrace/test.d/ftrace/fgraph-multi.tc | 103 ++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tc
tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tc new file +103 -0
@@ -0,0 +1,103 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: ftrace - function graph filters +# requires: set_ftrace_filter function_graph:tracer + +# Make sure that function graph filtering works + +INSTANCE1="instances/test1_$$" +INSTANCE2="instances/test2_$$" +INSTANCE3="instances/test3_$$" + +WD=`pwd` + +do_reset() { + cd $WD + if [ -d $INSTANCE1 ]; then + echo nop > $INSTANCE1/current_tracer + rmdir $INSTANCE1 + fi + if [ -d $INSTANCE2 ]; then + echo nop > $INSTANCE2/current_tracer + rmdir $INSTANCE2 + fi + if [ -d $INSTANCE3 ]; then + echo nop > $INSTANCE3/current_tracer + rmdir $INSTANCE3 + fi +} + +mkdir $INSTANCE1 +if ! grep -q function_graph $INSTANCE1/available_tracers; then + echo "function_graph not allowed with instances" + rmdir $INSTANCE1 + exit_unsupported +fi + +mkdir $INSTANCE2 +mkdir $INSTANCE3 + +fail() { # msg + do_reset + echo $1 + exit_fail +} + +disable_tracing +clear_trace + +do_test() { + REGEX=$1 + TEST=$2 + + # filter something, schedule is always good + if ! echo "$REGEX" > set_ftrace_filter; then + fail "can not enable filter $REGEX" + fi + + echo > trace + echo function_graph > current_tracer + enable_tracing + sleep 1 + # search for functions (has "{" or ";" on the line) + echo 0 > tracing_on + count=`cat trace | grep -v '^#' | grep -e '{' -e ';' | grep -v "$TEST" | wc -l` + echo 1 > tracing_on + if [ $count -ne 0 ]; then + fail "Graph filtering not working by itself against $TEST?" + fi + + # Make sure we did find something + echo 0 > tracing_on + count=`cat trace | grep -v '^#' | grep -e '{' -e ';' | grep "$TEST" | wc -l` + echo 1 > tracing_on + if [ $count -eq 0 ]; then + fail "No traces found with $TEST?" + fi +} + +do_test '*sched*' 'sched' +cd $INSTANCE1 +do_test '*lock*' 'lock' +cd $WD +cd $INSTANCE2 +do_test '*rcu*' 'rcu' +cd $WD +cd $INSTANCE3 +echo function_graph > current_tracer + +sleep 1 +count=`cat trace | grep -v '^#' | grep -e '{' -e ';' | grep "$TEST" | wc -l` +if [ $count -eq 0 ]; then + fail "No traces found with all tracing?" +fi + +cd $WD +echo nop > current_tracer +echo nop > $INSTANCE1/current_tracer +echo nop > $INSTANCE2/current_tracer +echo nop > $INSTANCE3/current_tracer + +do_reset + +exit 0
Signature · 1 line
2.43.0
Steven Rostedt
maintainerMasami,
This series passed all my tests, are you comfortable with me pushing them to linux-next?
-- Steve
Quoted 333 lines · On Mon, 03 Jun 2024 15:07:04 -0400 Steven Rostedt <[email protected]> wrote:
This is a continuation of the function graph multi user code. I wrote a proof of concept back in 2019 of this code[1] and Masami started cleaning it up. I started from Masami's work v10 that can be found here:
https://lore.kernel.org/linux-trace-kernel/171509088006.162236.7227326999861366050.stgit@devnote2/
This is *only* the code that allows multiple users of function graph tracing. This is not the fprobe work that Masami is working to add on top of it. As Masami took my proof of concept, there was still several things I disliked about that code. Instead of having Masami clean it up even more, I decided to take over on just my code and change it up a bit.
Changes since v2: https://lore.kernel.org/linux-trace-kernel/[email protected]
- Added comments describing which hashes the append and intersect functions were used for.
- Replaced checks of (NULL or
EMPTY_HASH) withftrace_hash_empty()helper function.- Added check at the end of
intersect_hash()to convert the hash to EMPTY hash if it doesn't have any functions.- Renamed
compare_ops()toops_equal()and return boolean (inversed return value).- Broke out
__ftrace_hash_move_and_update_ops()to use in bothftrace_hash_move_and_update_ops()andftrace_hash_move_and_update_subops().Diff between last version at end of this email.
Masami Hiramatsu (Google) (3):
function_graph: Handle tail calls for stack unwinding
function_graph: Use a simple LRU forfgraph_arrayindex number
ftrace: Add multiple fgraph storage selftestSteven Rostedt (Google) (9):
ftrace: Add subops logic to allow one ops to manage many
ftrace: Allow subops filtering to be modified
function_graph: Add pid tracing back to function graph tracer
function_graph: Usefor_each_set_bit()in__ftrace_return_to_handler()
function_graph: Use bitmask to loop on fgraph entry
function_graph: Usestatic_calland branch to optimize entry function
function_graph: Usestatic_calland branch to optimize return function selftests/ftrace: Addfunction_graphtracer to func-filter-pid test selftests/ftrace: Add fgraph-multi.tc testSteven Rostedt (VMware) (15):
function_graph: Convertret_stackto a series of longs
fgraph: UseBUILD_BUG_ON()to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic aroundftrace_graph_entryand return ftrace/function_graph: Passfgraph_opsto function graph callbacks
ftrace: Allowfunction_graphtracer to be enabled in instances
ftrace: Allow ftrace startup flags to exist without dynamic ftrace
function_graph: Have the instances use their ownftrace_opsfor filtering
function_graph: Add "task variables" per task forfgraph_ops
function_graph: Moveset_graph_functiontests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implementfgraph_reserve_data()andfgraph_retrieve_data()
function_graph: Add selftest for passing local variables----
include/linux/ftrace.h | 43 +- include/linux/sched.h | 2 +- include/linux/trace_recursion.h | 39 - kernel/trace/fgraph.c | 1044 ++++++++++++++++---- kernel/trace/ftrace.c | 522 +++++++++- kernel/trace/ftrace_internal.h | 5 +- kernel/trace/trace.h | 94 +- kernel/trace/trace_functions.c | 8 + kernel/trace/trace_functions_graph.c | 96 +- kernel/trace/trace_irqsoff.c | 10 +- kernel/trace/trace_sched_wakeup.c | 10 +- kernel/trace/trace_selftest.c | 259 ++++- .../selftests/ftrace/test.d/ftrace/fgraph-multi.tc | 103 ++ .../ftrace/test.d/ftrace/func-filter-pid.tc | 27 +- 14 files changed, 1945 insertions(+), 317 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tckernel/trace/ftrace.c +64
-46@@ -3170,7 +3170,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command)/* Simply make a copy of @src and return it */ static struct ftrace_hash *copy_hash (struct ftrace_hash *src) { -if (!src || src ==EMPTY_HASH ) +if (ftrace_hash_empty (src))return EMPTY_HASH ;return alloc_and_copy_ftrace_hash (src->size_bits, src);@@ -3187,6 +3187,9 @@ static struct ftrace_hash *copy_hash(struct ftrace_hash *src)* * Otherwise, go through all of @new_hash and add anything that @hash * doesn't already have, to @hash. +* +* The filter_hash updates uses just the append_hash() function +* and the notrace_hash does not. */ static int append_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash) {@@ -3195,11 +3198,11 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)int i;/* An empty hash does everything */ -if (!*hash || *hash ==EMPTY_HASH ) +if (ftrace_hash_empty (*hash))return 0 ;/* If new_hash has everything make hash have everything */ -if (!new_hash || new_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3217,7 +3220,12 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)return 0 ; } -/* Add to @hash only those that are in both @new_hash1 and @new_hash2 */ +/* +* Add to @hash only those that are in both @new_hash1 and @new_hash2 +* +* The notrace_hash updates uses just the intersect_hash() function +* and the filter_hash does not. +*/ static int intersect_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash1,struct ftrace_hash *new_hash2) {@@ -3229,8 +3237,7 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_has* If new_hash1 or new_hash2 is the EMPTY_HASH then make the hash * empty as well as empty for notrace means none are notraced. */ -if (!new_hash1 || new_hash1 ==EMPTY_HASH || - !new_hash2 || new_hash2 ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash1) ||ftrace_hash_empty (new_hash2)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3245,6 +3252,11 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hasreturn -ENOMEM ; } } +/* If nothing intersects, make it the empty set */ +if (ftrace_hash_empty (*hash)) { +free_ftrace_hash (*hash); + *hash =EMPTY_HASH ; + }return 0 ; }@@ -3266,7 +3278,7 @@ static struct ftrace_hash *append_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash;@@ -3300,59 +3312,76 @@ static struct ftrace_hash *intersect_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash; } -/* Returns 0 on equal or non-zero on non-equal */ -static int compare_ops (struct ftrace_hash *A ,struct ftrace_hash *B ) +static bool ops_equal (struct ftrace_hash *A ,struct ftrace_hash *B ) {struct ftrace_func_entry *entry;int size;int i; -if (!A ||A ==EMPTY_HASH ) -return !(!B ||B ==EMPTY_HASH ); +if (ftrace_hash_empty (A )) +return ftrace_hash_empty (B ); -if (!B ||B ==EMPTY_HASH ) -return !(!A ||A ==EMPTY_HASH ); +if (ftrace_hash_empty (B )) +return ftrace_hash_empty (A );if (A ->count !=B ->count) -return 1 ; +return false; size =1 <<A ->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (entry, &A ->buckets[i], hlist) {if (!__ftrace_lookup_ip (B , entry->ip)) -return 1 ; +return false; } } -return 0 ; +return true; } -static int ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, -struct ftrace_hash **orig_hash, -struct ftrace_hash *hash, -int enable); +static void ftrace_ops_update_code (struct ftrace_ops *ops, +struct ftrace_ops_hash *old_hash); + +static int __ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, +struct ftrace_hash **orig_hash, +struct ftrace_hash *hash, +int enable) +{ +struct ftrace_ops_hash old_hash_ops; +struct ftrace_hash *old_hash; +int ret; + + old_hash = *orig_hash; + old_hash_ops.filter_hash = ops->func_hash->filter_hash; + old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; + ret =ftrace_hash_move (ops, enable, orig_hash, hash); +if (!ret) { +ftrace_ops_update_code (ops, &old_hash_ops); +free_ftrace_hash_rcu (old_hash); + } +return ret; +}static int ftrace_update_ops (struct ftrace_ops *ops,struct ftrace_hash *filter_hash,struct ftrace_hash *notrace_hash) {int ret; -if (compare_ops (filter_hash, ops->func_hash->filter_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, - filter_hash,1 ); +if (!ops_equal (filter_hash, ops->func_hash->filter_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, + filter_hash,1 );if (ret <0 )return ret; } -if (compare_ops (notrace_hash, ops->func_hash->notrace_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, - notrace_hash,0 ); +if (!ops_equal (notrace_hash, ops->func_hash->notrace_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, + notrace_hash,0 );if (ret <0 )return ret; }@@ -3438,8 +3467,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int* o If either notrace_hash is empty then the final stays empty * o Otherwise, the final is an intersection between the hashes */ -if (ops->func_hash->filter_hash ==EMPTY_HASH || - subops->func_hash->filter_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->filter_hash) || +ftrace_hash_empty (subops->func_hash->filter_hash)) { filter_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3454,8 +3483,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int } } -if (ops->func_hash->notrace_hash ==EMPTY_HASH || - subops->func_hash->notrace_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->notrace_hash) || +ftrace_hash_empty (subops->func_hash->notrace_hash)) { notrace_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3591,7 +3620,7 @@ static int ftrace_hash_move_and_update_subops(struct ftrace_ops *subops, }/* Move the hash over to the new hash */ - ret =ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable); + ret =__ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable);free_ftrace_hash (new_hash);@@ -4822,11 +4851,6 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, struct ftrace_hash *hash, int enable) { -struct ftrace_ops_hash old_hash_ops; -struct ftrace_hash *old_hash; -struct ftrace_ops *op; -int ret; -if (ops->flags &FTRACE_OPS_FL_SUBOP )return ftrace_hash_move_and_update_subops (ops, orig_hash, hash, enable);@@ -4838,6 +4862,8 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops,* it will not affect subops that share it. */ if (!(ops->flags &FTRACE_OPS_FL_ENABLED )) { +struct ftrace_ops *op; +/* Check if any other manager subops maps to this hash */ do_for_each_ftrace_op (op, ftrace_ops_list) {struct ftrace_ops *subops;@@ -4851,15 +4877,7 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, }while_for_each_ftrace_op (op); } - old_hash = *orig_hash; - old_hash_ops.filter_hash = ops->func_hash->filter_hash; - old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; - ret =ftrace_hash_move (ops, enable, orig_hash, hash); -if (!ret) { -ftrace_ops_update_code (ops, &old_hash_ops); -free_ftrace_hash_rcu (old_hash); - } -return ret; +return __ftrace_hash_move_and_update_ops (ops, orig_hash, hash, enable); }static bool module_exists (const char *module)
Quoted 5 lines · On Tue, 4 Jun 2024 08:18:50 -0400 Steven Rostedt <[email protected]> wrote:
Masami,
This series passed all my tests, are you comfortable with me pushing them to linux-next?
Yes, this series looks good to me too.
for the series.
Thank you!
Quoted 341 lines
-- Steve
Quoted 333 lines · On Mon, 03 Jun 2024 15:07:04 -0400 Steven Rostedt <[email protected]> wrote:
This is a continuation of the function graph multi user code. I wrote a proof of concept back in 2019 of this code[1] and Masami started cleaning it up. I started from Masami's work v10 that can be found here:
https://lore.kernel.org/linux-trace-kernel/171509088006.162236.7227326999861366050.stgit@devnote2/
This is *only* the code that allows multiple users of function graph tracing. This is not the fprobe work that Masami is working to add on top of it. As Masami took my proof of concept, there was still several things I disliked about that code. Instead of having Masami clean it up even more, I decided to take over on just my code and change it up a bit.
Changes since v2: https://lore.kernel.org/linux-trace-kernel/[email protected]
- Added comments describing which hashes the append and intersect functions were used for.
- Replaced checks of (NULL or
EMPTY_HASH) withftrace_hash_empty()helper function.- Added check at the end of
intersect_hash()to convert the hash to EMPTY hash if it doesn't have any functions.- Renamed
compare_ops()toops_equal()and return boolean (inversed return value).- Broke out
__ftrace_hash_move_and_update_ops()to use in bothftrace_hash_move_and_update_ops()andftrace_hash_move_and_update_subops().Diff between last version at end of this email.
Masami Hiramatsu (Google) (3):
function_graph: Handle tail calls for stack unwinding
function_graph: Use a simple LRU forfgraph_arrayindex number
ftrace: Add multiple fgraph storage selftestSteven Rostedt (Google) (9):
ftrace: Add subops logic to allow one ops to manage many
ftrace: Allow subops filtering to be modified
function_graph: Add pid tracing back to function graph tracer
function_graph: Usefor_each_set_bit()in__ftrace_return_to_handler()
function_graph: Use bitmask to loop on fgraph entry
function_graph: Usestatic_calland branch to optimize entry function
function_graph: Usestatic_calland branch to optimize return function selftests/ftrace: Addfunction_graphtracer to func-filter-pid test selftests/ftrace: Add fgraph-multi.tc testSteven Rostedt (VMware) (15):
function_graph: Convertret_stackto a series of longs
fgraph: UseBUILD_BUG_ON()to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic aroundftrace_graph_entryand return ftrace/function_graph: Passfgraph_opsto function graph callbacks
ftrace: Allowfunction_graphtracer to be enabled in instances
ftrace: Allow ftrace startup flags to exist without dynamic ftrace
function_graph: Have the instances use their ownftrace_opsfor filtering
function_graph: Add "task variables" per task forfgraph_ops
function_graph: Moveset_graph_functiontests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implementfgraph_reserve_data()andfgraph_retrieve_data()
function_graph: Add selftest for passing local variables----
include/linux/ftrace.h | 43 +- include/linux/sched.h | 2 +- include/linux/trace_recursion.h | 39 - kernel/trace/fgraph.c | 1044 ++++++++++++++++---- kernel/trace/ftrace.c | 522 +++++++++- kernel/trace/ftrace_internal.h | 5 +- kernel/trace/trace.h | 94 +- kernel/trace/trace_functions.c | 8 + kernel/trace/trace_functions_graph.c | 96 +- kernel/trace/trace_irqsoff.c | 10 +- kernel/trace/trace_sched_wakeup.c | 10 +- kernel/trace/trace_selftest.c | 259 ++++- .../selftests/ftrace/test.d/ftrace/fgraph-multi.tc | 103 ++ .../ftrace/test.d/ftrace/func-filter-pid.tc | 27 +- 14 files changed, 1945 insertions(+), 317 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tckernel/trace/ftrace.c +64
-46@@ -3170,7 +3170,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command)/* Simply make a copy of @src and return it */ static struct ftrace_hash *copy_hash (struct ftrace_hash *src) { -if (!src || src ==EMPTY_HASH ) +if (ftrace_hash_empty (src))return EMPTY_HASH ;return alloc_and_copy_ftrace_hash (src->size_bits, src);@@ -3187,6 +3187,9 @@ static struct ftrace_hash *copy_hash(struct ftrace_hash *src)* * Otherwise, go through all of @new_hash and add anything that @hash * doesn't already have, to @hash. +* +* The filter_hash updates uses just the append_hash() function +* and the notrace_hash does not. */ static int append_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash) {@@ -3195,11 +3198,11 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)int i;/* An empty hash does everything */ -if (!*hash || *hash ==EMPTY_HASH ) +if (ftrace_hash_empty (*hash))return 0 ;/* If new_hash has everything make hash have everything */ -if (!new_hash || new_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3217,7 +3220,12 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)return 0 ; } -/* Add to @hash only those that are in both @new_hash1 and @new_hash2 */ +/* +* Add to @hash only those that are in both @new_hash1 and @new_hash2 +* +* The notrace_hash updates uses just the intersect_hash() function +* and the filter_hash does not. +*/ static int intersect_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash1,struct ftrace_hash *new_hash2) {@@ -3229,8 +3237,7 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_has* If new_hash1 or new_hash2 is the EMPTY_HASH then make the hash * empty as well as empty for notrace means none are notraced. */ -if (!new_hash1 || new_hash1 ==EMPTY_HASH || - !new_hash2 || new_hash2 ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash1) ||ftrace_hash_empty (new_hash2)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3245,6 +3252,11 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hasreturn -ENOMEM ; } } +/* If nothing intersects, make it the empty set */ +if (ftrace_hash_empty (*hash)) { +free_ftrace_hash (*hash); + *hash =EMPTY_HASH ; + }return 0 ; }@@ -3266,7 +3278,7 @@ static struct ftrace_hash *append_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash;@@ -3300,59 +3312,76 @@ static struct ftrace_hash *intersect_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash; } -/* Returns 0 on equal or non-zero on non-equal */ -static int compare_ops (struct ftrace_hash *A ,struct ftrace_hash *B ) +static bool ops_equal (struct ftrace_hash *A ,struct ftrace_hash *B ) {struct ftrace_func_entry *entry;int size;int i; -if (!A ||A ==EMPTY_HASH ) -return !(!B ||B ==EMPTY_HASH ); +if (ftrace_hash_empty (A )) +return ftrace_hash_empty (B ); -if (!B ||B ==EMPTY_HASH ) -return !(!A ||A ==EMPTY_HASH ); +if (ftrace_hash_empty (B )) +return ftrace_hash_empty (A );if (A ->count !=B ->count) -return 1 ; +return false; size =1 <<A ->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (entry, &A ->buckets[i], hlist) {if (!__ftrace_lookup_ip (B , entry->ip)) -return 1 ; +return false; } } -return 0 ; +return true; } -static int ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, -struct ftrace_hash **orig_hash, -struct ftrace_hash *hash, -int enable); +static void ftrace_ops_update_code (struct ftrace_ops *ops, +struct ftrace_ops_hash *old_hash); + +static int __ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, +struct ftrace_hash **orig_hash, +struct ftrace_hash *hash, +int enable) +{ +struct ftrace_ops_hash old_hash_ops; +struct ftrace_hash *old_hash; +int ret; + + old_hash = *orig_hash; + old_hash_ops.filter_hash = ops->func_hash->filter_hash; + old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; + ret =ftrace_hash_move (ops, enable, orig_hash, hash); +if (!ret) { +ftrace_ops_update_code (ops, &old_hash_ops); +free_ftrace_hash_rcu (old_hash); + } +return ret; +}static int ftrace_update_ops (struct ftrace_ops *ops,struct ftrace_hash *filter_hash,struct ftrace_hash *notrace_hash) {int ret; -if (compare_ops (filter_hash, ops->func_hash->filter_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, - filter_hash,1 ); +if (!ops_equal (filter_hash, ops->func_hash->filter_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, + filter_hash,1 );if (ret <0 )return ret; } -if (compare_ops (notrace_hash, ops->func_hash->notrace_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, - notrace_hash,0 ); +if (!ops_equal (notrace_hash, ops->func_hash->notrace_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, + notrace_hash,0 );if (ret <0 )return ret; }@@ -3438,8 +3467,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int* o If either notrace_hash is empty then the final stays empty * o Otherwise, the final is an intersection between the hashes */ -if (ops->func_hash->filter_hash ==EMPTY_HASH || - subops->func_hash->filter_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->filter_hash) || +ftrace_hash_empty (subops->func_hash->filter_hash)) { filter_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3454,8 +3483,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int } } -if (ops->func_hash->notrace_hash ==EMPTY_HASH || - subops->func_hash->notrace_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->notrace_hash) || +ftrace_hash_empty (subops->func_hash->notrace_hash)) { notrace_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3591,7 +3620,7 @@ static int ftrace_hash_move_and_update_subops(struct ftrace_ops *subops, }/* Move the hash over to the new hash */ - ret =ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable); + ret =__ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable);free_ftrace_hash (new_hash);@@ -4822,11 +4851,6 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, struct ftrace_hash *hash, int enable) { -struct ftrace_ops_hash old_hash_ops; -struct ftrace_hash *old_hash; -struct ftrace_ops *op; -int ret; -if (ops->flags &FTRACE_OPS_FL_SUBOP )return ftrace_hash_move_and_update_subops (ops, orig_hash, hash, enable);@@ -4838,6 +4862,8 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops,* it will not affect subops that share it. */ if (!(ops->flags &FTRACE_OPS_FL_ENABLED )) { +struct ftrace_ops *op; +/* Check if any other manager subops maps to this hash */ do_for_each_ftrace_op (op, ftrace_ops_list) {struct ftrace_ops *subops;@@ -4851,15 +4877,7 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, }while_for_each_ftrace_op (op); } - old_hash = *orig_hash; - old_hash_ops.filter_hash = ops->func_hash->filter_hash; - old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; - ret =ftrace_hash_move (ops, enable, orig_hash, hash); -if (!ret) { -ftrace_ops_update_code (ops, &old_hash_ops); -free_ftrace_hash_rcu (old_hash); - } -return ret; +return __ftrace_hash_move_and_update_ops (ops, orig_hash, hash, enable); }static bool module_exists (const char *module)
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Mark Rutland
reviewerHi Steve, Masami,
Quoted 5 lines · Steven Rostedt
Masami,
This series passed all my tests, are you comfortable with me pushing them to linux-next?
As a heads-up (and not to block pushing this into next), I just gave this a spin on arm64 atop v6.10-rc2, and running the selftests I see:
ftrace - function pid filters
(instance) ftrace - function pid filters
... both go from [PASS] to [FAIL].
Everything else looks good -- I'll go dig into why that's happening.
It's possible that's just something odd with the filesystem I'm using (e.g. the wnership test failed because this lacks 'stat').
Mark.
Quoted 341 lines
-- Steve
Quoted 333 lines · On Mon, 03 Jun 2024 15:07:04 -0400 Steven Rostedt <[email protected]> wrote:
This is a continuation of the function graph multi user code. I wrote a proof of concept back in 2019 of this code[1] and Masami started cleaning it up. I started from Masami's work v10 that can be found here:
https://lore.kernel.org/linux-trace-kernel/171509088006.162236.7227326999861366050.stgit@devnote2/
This is *only* the code that allows multiple users of function graph tracing. This is not the fprobe work that Masami is working to add on top of it. As Masami took my proof of concept, there was still several things I disliked about that code. Instead of having Masami clean it up even more, I decided to take over on just my code and change it up a bit.
Changes since v2: https://lore.kernel.org/linux-trace-kernel/[email protected]
- Added comments describing which hashes the append and intersect functions were used for.
- Replaced checks of (NULL or
EMPTY_HASH) withftrace_hash_empty()helper function.- Added check at the end of
intersect_hash()to convert the hash to EMPTY hash if it doesn't have any functions.- Renamed
compare_ops()toops_equal()and return boolean (inversed return value).- Broke out
__ftrace_hash_move_and_update_ops()to use in bothftrace_hash_move_and_update_ops()andftrace_hash_move_and_update_subops().Diff between last version at end of this email.
Masami Hiramatsu (Google) (3):
function_graph: Handle tail calls for stack unwinding
function_graph: Use a simple LRU forfgraph_arrayindex number
ftrace: Add multiple fgraph storage selftestSteven Rostedt (Google) (9):
ftrace: Add subops logic to allow one ops to manage many
ftrace: Allow subops filtering to be modified
function_graph: Add pid tracing back to function graph tracer
function_graph: Usefor_each_set_bit()in__ftrace_return_to_handler()
function_graph: Use bitmask to loop on fgraph entry
function_graph: Usestatic_calland branch to optimize entry function
function_graph: Usestatic_calland branch to optimize return function selftests/ftrace: Addfunction_graphtracer to func-filter-pid test selftests/ftrace: Add fgraph-multi.tc testSteven Rostedt (VMware) (15):
function_graph: Convertret_stackto a series of longs
fgraph: UseBUILD_BUG_ON()to make sure we have structures divisible by long
function_graph: Add an array structure that will allow multiple callbacks
function_graph: Allow multiple users to attach to function graph
function_graph: Remove logic aroundftrace_graph_entryand return ftrace/function_graph: Passfgraph_opsto function graph callbacks
ftrace: Allowfunction_graphtracer to be enabled in instances
ftrace: Allow ftrace startup flags to exist without dynamic ftrace
function_graph: Have the instances use their ownftrace_opsfor filtering
function_graph: Add "task variables" per task forfgraph_ops
function_graph: Moveset_graph_functiontests to shadow stack global var
function_graph: Move graph depth stored data to shadow stack global var
function_graph: Move graph notrace bit to shadow stack global var
function_graph: Implementfgraph_reserve_data()andfgraph_retrieve_data()
function_graph: Add selftest for passing local variables----
include/linux/ftrace.h | 43 +- include/linux/sched.h | 2 +- include/linux/trace_recursion.h | 39 - kernel/trace/fgraph.c | 1044 ++++++++++++++++---- kernel/trace/ftrace.c | 522 +++++++++- kernel/trace/ftrace_internal.h | 5 +- kernel/trace/trace.h | 94 +- kernel/trace/trace_functions.c | 8 + kernel/trace/trace_functions_graph.c | 96 +- kernel/trace/trace_irqsoff.c | 10 +- kernel/trace/trace_sched_wakeup.c | 10 +- kernel/trace/trace_selftest.c | 259 ++++- .../selftests/ftrace/test.d/ftrace/fgraph-multi.tc | 103 ++ .../ftrace/test.d/ftrace/func-filter-pid.tc | 27 +- 14 files changed, 1945 insertions(+), 317 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/fgraph-multi.tckernel/trace/ftrace.c +64
-46@@ -3170,7 +3170,7 @@ int ftrace_shutdown(struct ftrace_ops *ops, int command)/* Simply make a copy of @src and return it */ static struct ftrace_hash *copy_hash (struct ftrace_hash *src) { -if (!src || src ==EMPTY_HASH ) +if (ftrace_hash_empty (src))return EMPTY_HASH ;return alloc_and_copy_ftrace_hash (src->size_bits, src);@@ -3187,6 +3187,9 @@ static struct ftrace_hash *copy_hash(struct ftrace_hash *src)* * Otherwise, go through all of @new_hash and add anything that @hash * doesn't already have, to @hash. +* +* The filter_hash updates uses just the append_hash() function +* and the notrace_hash does not. */ static int append_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash) {@@ -3195,11 +3198,11 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)int i;/* An empty hash does everything */ -if (!*hash || *hash ==EMPTY_HASH ) +if (ftrace_hash_empty (*hash))return 0 ;/* If new_hash has everything make hash have everything */ -if (!new_hash || new_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3217,7 +3220,12 @@ static int append_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hash)return 0 ; } -/* Add to @hash only those that are in both @new_hash1 and @new_hash2 */ +/* +* Add to @hash only those that are in both @new_hash1 and @new_hash2 +* +* The notrace_hash updates uses just the intersect_hash() function +* and the filter_hash does not. +*/ static int intersect_hash (struct ftrace_hash **hash,struct ftrace_hash *new_hash1,struct ftrace_hash *new_hash2) {@@ -3229,8 +3237,7 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_has* If new_hash1 or new_hash2 is the EMPTY_HASH then make the hash * empty as well as empty for notrace means none are notraced. */ -if (!new_hash1 || new_hash1 ==EMPTY_HASH || - !new_hash2 || new_hash2 ==EMPTY_HASH ) { +if (ftrace_hash_empty (new_hash1) ||ftrace_hash_empty (new_hash2)) {free_ftrace_hash (*hash); *hash =EMPTY_HASH ;return 0 ;@@ -3245,6 +3252,11 @@ static int intersect_hash(struct ftrace_hash **hash, struct ftrace_hash *new_hasreturn -ENOMEM ; } } +/* If nothing intersects, make it the empty set */ +if (ftrace_hash_empty (*hash)) { +free_ftrace_hash (*hash); + *hash =EMPTY_HASH ; + }return 0 ; }@@ -3266,7 +3278,7 @@ static struct ftrace_hash *append_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash;@@ -3300,59 +3312,76 @@ static struct ftrace_hash *intersect_hashes(struct ftrace_ops *ops)return NULL ; }/* Nothing more to do if new_hash is empty */ -if (new_hash ==EMPTY_HASH ) +if (ftrace_hash_empty (new_hash))break ; }return new_hash; } -/* Returns 0 on equal or non-zero on non-equal */ -static int compare_ops (struct ftrace_hash *A ,struct ftrace_hash *B ) +static bool ops_equal (struct ftrace_hash *A ,struct ftrace_hash *B ) {struct ftrace_func_entry *entry;int size;int i; -if (!A ||A ==EMPTY_HASH ) -return !(!B ||B ==EMPTY_HASH ); +if (ftrace_hash_empty (A )) +return ftrace_hash_empty (B ); -if (!B ||B ==EMPTY_HASH ) -return !(!A ||A ==EMPTY_HASH ); +if (ftrace_hash_empty (B )) +return ftrace_hash_empty (A );if (A ->count !=B ->count) -return 1 ; +return false; size =1 <<A ->size_bits;for (i =0 ; i < size; i++) {hlist_for_each_entry (entry, &A ->buckets[i], hlist) {if (!__ftrace_lookup_ip (B , entry->ip)) -return 1 ; +return false; } } -return 0 ; +return true; } -static int ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, -struct ftrace_hash **orig_hash, -struct ftrace_hash *hash, -int enable); +static void ftrace_ops_update_code (struct ftrace_ops *ops, +struct ftrace_ops_hash *old_hash); + +static int __ftrace_hash_move_and_update_ops (struct ftrace_ops *ops, +struct ftrace_hash **orig_hash, +struct ftrace_hash *hash, +int enable) +{ +struct ftrace_ops_hash old_hash_ops; +struct ftrace_hash *old_hash; +int ret; + + old_hash = *orig_hash; + old_hash_ops.filter_hash = ops->func_hash->filter_hash; + old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; + ret =ftrace_hash_move (ops, enable, orig_hash, hash); +if (!ret) { +ftrace_ops_update_code (ops, &old_hash_ops); +free_ftrace_hash_rcu (old_hash); + } +return ret; +}static int ftrace_update_ops (struct ftrace_ops *ops,struct ftrace_hash *filter_hash,struct ftrace_hash *notrace_hash) {int ret; -if (compare_ops (filter_hash, ops->func_hash->filter_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, - filter_hash,1 ); +if (!ops_equal (filter_hash, ops->func_hash->filter_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->filter_hash, + filter_hash,1 );if (ret <0 )return ret; } -if (compare_ops (notrace_hash, ops->func_hash->notrace_hash)) { - ret =ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, - notrace_hash,0 ); +if (!ops_equal (notrace_hash, ops->func_hash->notrace_hash)) { + ret =__ftrace_hash_move_and_update_ops (ops, &ops->func_hash->notrace_hash, + notrace_hash,0 );if (ret <0 )return ret; }@@ -3438,8 +3467,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int* o If either notrace_hash is empty then the final stays empty * o Otherwise, the final is an intersection between the hashes */ -if (ops->func_hash->filter_hash ==EMPTY_HASH || - subops->func_hash->filter_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->filter_hash) || +ftrace_hash_empty (subops->func_hash->filter_hash)) { filter_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3454,8 +3483,8 @@ int ftrace_startup_subops(struct ftrace_ops *ops, struct ftrace_ops *subops, int } } -if (ops->func_hash->notrace_hash ==EMPTY_HASH || - subops->func_hash->notrace_hash ==EMPTY_HASH ) { +if (ftrace_hash_empty (ops->func_hash->notrace_hash) || +ftrace_hash_empty (subops->func_hash->notrace_hash)) { notrace_hash =EMPTY_HASH ; }else { size_bits = max(ops->func_hash->filter_hash->size_bits,@@ -3591,7 +3620,7 @@ static int ftrace_hash_move_and_update_subops(struct ftrace_ops *subops, }/* Move the hash over to the new hash */ - ret =ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable); + ret =__ftrace_hash_move_and_update_ops (ops, orig_hash, new_hash, enable);free_ftrace_hash (new_hash);@@ -4822,11 +4851,6 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, struct ftrace_hash *hash, int enable) { -struct ftrace_ops_hash old_hash_ops; -struct ftrace_hash *old_hash; -struct ftrace_ops *op; -int ret; -if (ops->flags &FTRACE_OPS_FL_SUBOP )return ftrace_hash_move_and_update_subops (ops, orig_hash, hash, enable);@@ -4838,6 +4862,8 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops,* it will not affect subops that share it. */ if (!(ops->flags &FTRACE_OPS_FL_ENABLED )) { +struct ftrace_ops *op; +/* Check if any other manager subops maps to this hash */ do_for_each_ftrace_op (op, ftrace_ops_list) {struct ftrace_ops *subops;@@ -4851,15 +4877,7 @@ static int ftrace_hash_move_and_update_ops(struct ftrace_ops *ops, }while_for_each_ftrace_op (op); } - old_hash = *orig_hash; - old_hash_ops.filter_hash = ops->func_hash->filter_hash; - old_hash_ops.notrace_hash = ops->func_hash->notrace_hash; - ret =ftrace_hash_move (ops, enable, orig_hash, hash); -if (!ret) { -ftrace_ops_update_code (ops, &old_hash_ops); -free_ftrace_hash_rcu (old_hash); - } -return ret; +return __ftrace_hash_move_and_update_ops (ops, orig_hash, hash, enable); }static bool module_exists (const char *module)
Steven Rostedt
maintainerQuoted 15 lines · On Tue, 4 Jun 2024 15:44:40 +0100 Mark Rutland <[email protected]> wrote:
Hi Steve, Masami,
Quoted 5 lines · Steven Rostedt
Masami,
This series passed all my tests, are you comfortable with me pushing them to linux-next?
As a heads-up (and not to block pushing this into next), I just gave this a spin on arm64 atop v6.10-rc2, and running the selftests I see:
ftrace - function pid filters
(instance) ftrace - function pid filters
Quoted 6 lines
... both go from [PASS] to [FAIL].
Everything else looks good -- I'll go dig into why that's happening.
It's possible that's just something odd with the filesystem I'm using (e.g. the wnership test failed because this lacks 'stat').
Thanks for the update. I could be something I missed in patch 13 that had to put back the pid code.
There may have been something arch specific that I'm unaware about. I'll look at that deeper.
-- Steve
Mark Rutland
reviewerQuoted 22 lines · Steven Rostedt
Quoted 19 lines · On Tue, 4 Jun 2024 15:44:40 +0100 Mark Rutland <[email protected]> wrote:
Hi Steve, Masami,
Quoted 5 lines · Steven Rostedt
Masami,
This series passed all my tests, are you comfortable with me pushing them to linux-next?
As a heads-up (and not to block pushing this into next), I just gave this a spin on arm64 atop v6.10-rc2, and running the selftests I see:
ftrace - function pid filters
(instance) ftrace - function pid filters... both go from [PASS] to [FAIL].
Everything else looks good -- I'll go dig into why that's happening.
Quoted 8 lines
Quoted 2 lines
It's possible that's just something odd with the filesystem I'm using (e.g. the wnership test failed because this lacks 'stat').
Thanks for the update. I could be something I missed in patch 13 that had to put back the pid code.
There may have been something arch specific that I'm unaware about. I'll look at that deeper.
It looks like e are lines in the trace that it doesn't expect:
+ cat trace
+ grep -v ^#
+ grep 970
+ wc -l
+ count_pid=0
+ cat trace
+ grep -v ^#
+ grep -v 970
+ wc -l
+ count_other=3
+ [ 0 -eq 0 -o 3 -ne 0 ]
+ fail PID filtering not working?
... where we expect that count_other to be 0.
I hacked in a 'cat trace' just before the 'fail' and that shows:
+ cat trace
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) ! 143.685 us | kernel_clone();
3) ! 127.055 us | kernel_clone();
1) ! 127.170 us | kernel_clone();
3) ! 126.840 us | kernel_clone();
I'm not sure if that's legitimate output the test is failing to account for or if that indicates a kernel-side issue.
Mark.
Steven Rostedt
maintainerQuoted 32 lines · On Tue, 4 Jun 2024 18:04:22 +0100 Mark Rutland <[email protected]> wrote:
Quoted 2 lines
There may have been something arch specific that I'm unaware about. I'll look at that deeper.
It looks like e are lines in the trace that it doesn't expect:
+ cat trace
+ grep -v ^#
+ grep 970
+ wc -l
+ count_pid=0
+ cat trace
+ grep -v ^#
+ grep -v 970
+ wc -l
+ count_other=3
+ [ 0 -eq 0 -o 3 -ne 0 ]
+ fail PID filtering not working?... where we expect that
count_otherto be 0.I hacked in a 'cat trace' just before the 'fail' and that shows:
+ cat trace
# tracer:function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) ! 143.685 us |kernel_clone();
3) ! 127.055 us |kernel_clone();
1) ! 127.170 us |kernel_clone();
3) ! 126.840 us |kernel_clone();
Quoted 2 lines
I'm not sure if that's legitimate output the test is failing to account for or if that indicates a kernel-side issue.
Bah, I just ran the test.d/ftrace/func-filter-pid.tc and it fails too. This did pass my other tests that do run ftracetests. Hmm, I just ran it on my test box that does the tests and it passes there. I wonder if there's some config option that makes it fail :-/
Well, now that I see it fail, I can investigate.
-- Steve
Steven Rostedt
maintainerQuoted 6 lines · On Tue, 4 Jun 2024 14:57:42 -0400 Steven Rostedt <[email protected]> wrote:
Bah, I just ran the test.d/ftrace/func-filter-pid.tc and it fails too. This did pass my other tests that do run ftracetests. Hmm, I just ran it on my test box that does the tests and it passes there. I wonder if there's some config option that makes it fail :-/
Well, now that I see it fail, I can investigate.
Ah, figured it out. The updated pid test isn't working. This explains why my test machine didn't fail, as it doesn't have the updated ftracetests.
The problem was that I never set the funcgraph-proc option, even though I saved it :-p
That is, it shows:
Quoted 9 lines
+ cat trace
# tracer:function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
3) ! 143.685 us |kernel_clone();
3) ! 127.055 us |kernel_clone();
1) ! 127.170 us |kernel_clone();
3) ! 126.840 us |kernel_clone();
But when you do: echo 1 > options/funcgraph-proc
You get:
# cat trace # tracer: function_graph # # CPU TASK/PID DURATION FUNCTION CALLS # | | | | | | | | | 4) bash-939 | # 1070.009 us | kernel_clone(); 4) bash-939 | # 1116.903 us | kernel_clone(); 5) bash-939 | ! 976.133 us | kernel_clone(); 5) bash-939 | ! 954.012 us | kernel_clone(); 5) bash-939 | ! 905.825 us | kernel_clone(); 5) bash-939 | # 1130.922 us | kernel_clone(); 7) bash-939 | # 1097.648 us | kernel_clone(); 0) bash-939 | # 1008.000 us | kernel_clone(); 3) bash-939 | # 1023.391 us | kernel_clone(); 4) bash-939 | # 1033.008 us | kernel_clone(); 4) bash-939 | ! 949.072 us | kernel_clone(); 4) bash-939 | # 1027.990 us | kernel_clone(); 4) bash-939 | ! 954.678 us | kernel_clone(); 4) bash-939 | ! 996.557 us | kernel_clone();
Without that option, function graph does no show what process is being recorded (except at sched switch)
Can you add this patch to the test and see if it works again?
-- Steve
tools/testing/selftests/ftrace/test.d/ftrace/func-filter-pid.tc +2 -0
@@ -8,6 +8,7 @@ # Also test it on an instance directory do_function_fork=1 +do_funcgraph_proc=1 if [ ! -f options/function-fork ]; then do_function_fork=0
@@ -28,6 +29,7 @@ fi if [ $do_funcgraph_proc -eq 1 ]; then orig_value2=`cat options/funcgraph-proc` + echo 1 > options/funcgraph-proc fi do_reset() {
Mark Rutland
reviewerQuoted 5 lines · Steven Rostedt
This is a continuation of the function graph multi user code. I wrote a proof of concept back in 2019 of this code[1] and Masami started cleaning it up. I started from Masami's work v10 that can be found here:
Quoted 8 lines
https://lore.kernel.org/linux-trace-kernel/171509088006.162236.7227326999861366050.stgit@devnote2/
This is *only* the code that allows multiple users of function graph tracing. This is not the fprobe work that Masami is working to add on top of it. As Masami took my proof of concept, there was still several things I disliked about that code. Instead of having Masami clean it up even more, I decided to take over on just my code and change it up a bit.
FWIW, this is useful to me as-is for testing stacktracing.
Before this series I had some horrid hacks to manipulate the global filters, and after this series I just need to manipulate the filters in fgraph_ops::ops, which is *much* nicer.
I've pushed out my WIP to:
https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=stacktrace/tests
... which is all to say, this is useful and I've given this some testing beyond the usual ftrace tests.
Mark.
Steven Rostedt
maintainerRe: [PATCH v3 13/27] function_graph: Add pid tracing back to function graph tracer
In reply to Steven RostedtQuoted 11 lines · On Mon, 03 Jun 2024 15:07:17 -0400 Steven Rostedt <[email protected]> wrote:
+++
b/kernel/trace/ftrace.ckernel/trace/ftrace.c +1
-1@@ -100,7 +100,7 @@ struct ftrace_ops *function_trace_op __read_mostly = &ftrace_list_end;/* What to set function_trace_op to */ static struct ftrace_ops *set_function_trace_op; -static bool ftrace_pids_enabled (struct ftrace_ops *ops) +bool ftrace_pids_enabled (struct ftrace_ops *ops) {struct trace_array *tr;
kernel/trace/ftrace.c static void ftrace_update_pid_func(void) · line 405
Quoted 4 lines
kernel/trace/ftrace.c +0
-1@@ -402,10 +402,11 @@ static void ftrace_update_pid_func(void)if (op->flags &FTRACE_OPS_FL_PID ) { op->func =ftrace_pids_enabled (op) ? ftrace_pid_func : op->saved_func; -ftrace_update_trampoline (op);
Bah, this patch accidentally removed the above function and broke pid tracing. Hmm, not sure why this still passed the tests. Will investigate.
-- Steve
Quoted 8 lines
kernel/trace/ftrace.c +2
-0} }while_for_each_ftrace_op (op); +fgraph_update_pid_func (); +update_ftrace_function (); }