ftrace: Add function arguments to function tracers
Subsystems (4)
Steven Rostedt
maintainerv2 cover ftrace: Add function arguments to function tracers
These patches add support for printing function arguments in ftrace.
Example usage:
function tracer:
~# cd /sys/kernel/tracing/
~# echo icmp_rcv >set_ftrace_filter
~# echo function >current_tracer
~# echo 1 >options/func-args
~# ping -c 10 127.0.0.1
[..]
~# cat trace
[..]
ping-1277 [030] ..s1. 39.120939: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 39.120946: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 40.179724: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 40.179730: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 41.219700: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 41.219706: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 42.259717: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 42.259725: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 43.299735: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu
ping-1277 [030] ..s1. 43.299742: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcufunction graph:
~# cd /sys/kernel/tracing
~# echo icmp_rcv >set_graph_function
~# echo function_graph >current_tracer
~# echo 1 >options/funcgraph-args
~# ping -c 1 127.0.0.1
~# cat trace
30) | icmp_rcv(skb=0xa0ecab00) {
30) | __skb_checksum_complete(skb=0xa0ecab00) {
30) | skb_checksum(skb=0xa0ecab00, offset=0, len=64, csum=0) {
30) | __skb_checksum(skb=0xa0ecab00, offset=0, len=64, csum=0, ops=0x232e0327a88) {
30) 0.418 us | csum_partial(buff=0xa0d20924, len=64, sum=0)
30) 0.985 us | }
30) 1.463 us | }
30) 2.039 us | }[..]
This was last posted by Sven Schnelle here:
https://lore.kernel.org/all/[email protected]/
As Sven hasn't worked on it since, I decided to continue to push it through. I'm keeping Sven as original author and added myself as "Co-developed-by".
The main changes are:
- Made the kconfig option unconditional if all the dependencies are set.
- Not save ftrace_regs in the ring buffer, as that is an abstract descriptor defined by the architectures and should remain opaque from generic code. Instead, the args are read at the time they are recorded (with the ftrace_regs passed to the callback function), and saved into the ring buffer. Then the print function only takes an array of elements.
This could allow archs to retrieve arguments that are on the stack where as, post processing ftrace_regs could cause undesirable results.
- Made the function and function graph entry events dynamically sized to allow the arguments to be appended to the event in the ring buffer. The print function only looks to see if the event saved in the ring buffer is big enough to hold all the arguments defined by the new FTRACE_REGS_MAX_ARGS macro and if so, it will assume there are arguments there and print them. This also means user space will not break on reading these events as arguments will simply be ignored.
- The printing of the arguments has some more data when things are not processed by BPF. Any unsupported argument will have the type printed out in the ring buffer.
- Also removed the spaces around the '=' as that's more in line to how trace events show their fields.
- One new patch I added to convert function graph tracing over to using args as soon as the user sets the option even if function graph tracing is enabled. Function tracer did this already by default.
Steven Rostedt (1):
ftrace: Have funcgraph-args take affect during tracing
Sven Schnelle (3):
ftrace: Add print_function_args()
ftrace: Add support for function argument to graph tracer
ftrace: Add arguments to function tracer
----
include/linux/ftrace_regs.h | 5 + kernel/trace/Kconfig | 12 +++ kernel/trace/trace.c | 12 ++- kernel/trace/trace.h | 4 +- kernel/trace/trace_entries.h | 12 ++- kernel/trace/trace_functions.c | 46 ++++++++- kernel/trace/trace_functions_graph.c | 174 ++++++++++++++++++++++++++++------- kernel/trace/trace_irqsoff.c | 4 +- kernel/trace/trace_output.c | 96 ++++++++++++++++++- kernel/trace/trace_output.h | 9 ++ kernel/trace/trace_sched_wakeup.c | 4 +- 11 files changed, 324 insertions(+), 54 deletions(-)
Steven Rostedt
maintainerPATCH v2 1/4 ftrace: Add print_function_args()
From: Sven Schnelle <[email protected]>
Add a function to decode argument types with the help of BTF. Will be used to display arguments in the function and function graph tracer.
It can only handle simply arguments and up to FTRACE_REGS_MAX_ARGS number of arguments. When it hits a max, it will print ", ...":
page_to_skb(vi=0xffff8d53842dc980, rq=0xffff8d53843a0800, page=0xfffffc2e04337c00, offset=6160, len=64, truesize=1536, ...)
And if it hits an argument that is not recognized, it will print the raw value and the type of argument it is:
make_vfsuid(idmap=0xffffffff87f99db8, fs_userns=0xffffffff87e543c0, kuid=0x0 (STRUCT)) __pti_set_user_pgtbl(pgdp=0xffff8d5384ab47f8, pgd=0x110e74067 (STRUCT))
---
Changes since v1: https://lore.kernel.org/[email protected]
- Added Config option FUNCTION_TRACE_ARGS to this patch (unconditional if dependencies are met)
- Changed the print_function_args() function to take an array of unsigned long args and not the ftrace_regs pointer. The ftrace_regs should be opaque from generic code.
- Have the function print the name of an BTF type that is not supported.
- Added FTRACE_REGS_MAX_ARGS as the number of arguments saved in the event and printed out.
- Print "...," if the number of arguments goes past FTRACE_REGS_MAX_ARGS.
include/linux/ftrace_regs.h | 5 +++ kernel/trace/Kconfig | 6 +++ kernel/trace/trace_output.c | 78 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace_output.h | 9 +++++ 4 files changed, 98 insertions(+)
include/linux/ftrace_regs.h +5 -0
@@ -35,4 +35,9 @@ struct ftrace_regs;#endif /* HAVE_ARCH_FTRACE_REGS */ +/* This can be overridden by the architectures */ +#ifndef FTRACE_REGS_MAX_ARGS +# define FTRACE_REGS_MAX_ARGS 6 +#endif +#endif /* _LINUX_FTRACE_REGS_H */
kernel/trace/Kconfig +6 -0
@@ -263,6 +263,12 @@ config FUNCTION_GRAPH_RETADDR the function is called. This feature is off by default, and you can enable it via the trace option funcgraph-retaddr. +config FUNCTION_TRACE_ARGS + bool + depends on HAVE_FUNCTION_ARG_ACCESS_API + depends on DEBUG_INFO_BTF + default y + config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically" depends on FUNCTION_TRACER
kernel/trace/trace_output.c +78 -0
92 lines not on this page · raw message
kernel/trace/trace_output.h +9 -0
@@ -41,5 +41,14 @@ extern struct rw_semaphore trace_event_sem;#define SEQ_PUT_HEX_FIELD (s, x) \ trace_seq_putmem_hex(s, &(x), sizeof(x)) +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func); +#else +static inline void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) { +trace_seq_puts (s,"()" ); +} +#endif #endif
Signature · 1 line
2.45.2
Steven Rostedt
maintainerPATCH v2 2/4 ftrace: Add support for function argument to graph tracer
In reply to Steven RostedtFrom: Sven Schnelle <[email protected]>
Wire up the code to print function arguments in the function graph tracer. This functionality can be enabled/disabled during runtime with options/funcgraph-args.
Example usage:
6) | dummy_xmit [dummy](skb = 0x8887c100, dev = 0x872ca000) {
6) | consume_skb(skb = 0x8887c100) {
6) | skb_release_head_state(skb = 0x8887c100) {
6) 0.178 us | sock_wfree(skb = 0x8887c100)
6) 0.627 us | }---
Changes since v1: https://lore.kernel.org/[email protected]
- Record just FTRACE_REGS_MAX_ARGS into the ring buffer and not the entire ftrace_regs structure, as that structure should be opaque from generic code.
- Make the function graph entry event have a dynamic size, so that an array of arguments may be added in the ring buffer after it.
kernel/trace/Kconfig | 6 ++ kernel/trace/trace.h | 1 + kernel/trace/trace_entries.h | 7 +- kernel/trace/trace_functions_graph.c | 147 +++++++++++++++++++++------ 4 files changed, 125 insertions(+), 36 deletions(-)
kernel/trace/Kconfig +6 -0
@@ -268,6 +268,12 @@ config FUNCTION_TRACE_ARGS depends on HAVE_FUNCTION_ARG_ACCESS_API depends on DEBUG_INFO_BTF default y + help + If supported with function argument access API and BTF, then + the function tracer and function graph tracer will support printing + of function arguments. This feature is off by default, and can be + enabled via the trace option func-args (for the function tracer) and + funcgraph-args (for the function graph tracer) config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically"
kernel/trace/trace.h +1 -0
@@ -887,6 +887,7 @@ static __always_inline bool ftrace_hash_empty(struct ftrace_hash *hash)#define TRACE_GRAPH_PRINT_RETVAL 0x800#define TRACE_GRAPH_PRINT_RETVAL_HEX 0x1000#define TRACE_GRAPH_PRINT_RETADDR 0x2000 +#define TRACE_GRAPH_ARGS 0x4000#define TRACE_GRAPH_PRINT_FILL_SHIFT 28#define TRACE_GRAPH_PRINT_FILL_MASK (0x3 << TRACE_GRAPH_PRINT_FILL_SHIFT)
kernel/trace/trace_entries.h +4 -3
@@ -72,17 +72,18 @@ FTRACE_ENTRY_REG(function, ftrace_entry, );/* Function call entry */ -FTRACE_ENTRY_PACKED (funcgraph_entry, ftrace_graph_ent_entry, +FTRACE_ENTRY (funcgraph_entry, ftrace_graph_ent_entry,TRACE_GRAPH_ENT ,F_STRUCT ( __field_struct( struct ftrace_graph_ent, graph_ent ) __field_packed( unsigned long, graph_ent, func ) -__field_packed ( int, graph_ent, depth ) + __field_packed( unsigned long, graph_ent, depth ) + __dynamic_array(unsigned long, args ) ), -F_printk ("--> %ps (%d)" , (void *)__entry->func, __entry->depth) +F_printk ("--> %ps (%lu)" , (void *)__entry->func, __entry->depth) );#ifdef CONFIG_FUNCTION_GRAPH_RETADDR
kernel/trace/trace_functions_graph.c +114 -33
289 lines not on this page · raw message
Signature · 1 line
2.45.2
Steven Rostedt
maintainerPATCH v2 3/4 ftrace: Have funcgraph-args take affect during tracing
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Currently, when function_graph is started, it looks at the option funcgraph-args, and if it is set, it will enable tracing of the arguments.
But if tracing is already running, and the user enables funcgraph-args, it will have no effect. Instead, it should enable argument tracing when it is enabled, even if it means disabling the function graph tracing for a short time in order to do the transition.
--- kernel/trace/trace_functions_graph.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)
kernel/trace/trace_functions_graph.c +26 -1
@@ -469,7 +469,7 @@ static int graph_trace_init(struct trace_array *tr)else tr->gops->retfunc = trace_graph_return; -/* Make gops functions are visible before we start tracing */ +/* Make gops functions visible before we start tracing */ smp_mb (); ret =register_ftrace_graph (tr->gops);
@@ -480,6 +480,28 @@ static int graph_trace_init(struct trace_array *tr)return 0 ; } +static int ftrace_graph_trace_args (struct trace_array *tr,int set) +{ +trace_func_graph_ent_t entry; + +if (set) + entry = trace_graph_entry_args; +else + entry = trace_graph_entry; + +/* See if there's any changes */ +if (tr->gops->entryfunc == entry) +return 0 ; + +unregister_ftrace_graph (tr->gops); + + tr->gops->entryfunc = entry; + +/* Make gops functions visible before we start tracing */ +smp_mb (); +return register_ftrace_graph (tr->gops); +} +static void graph_trace_reset (struct trace_array *tr) {tracing_stop_cmdline_record ();
@@ -1609,6 +1631,9 @@ func_graph_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set)if (bit ==TRACE_GRAPH_GRAPH_TIME )ftrace_graph_graph_time_control (set); +if (bit ==TRACE_GRAPH_ARGS ) +return ftrace_graph_trace_args (tr, set); +return 0 ; }
Signature · 1 line
2.45.2
Steven Rostedt
maintainerPATCH v2 4/4 ftrace: Add arguments to function tracer
In reply to Steven RostedtFrom: Sven Schnelle <[email protected]>
Wire up the code to print function arguments in the function tracer. This functionality can be enabled/disabled during runtime with options/func-args.
ping-689 [004] b.... 77.170220: dummy_xmit(skb = 0x82904800, dev = 0x882d0000) <-dev_hard_start_xmit
---
Changes since v1: https://lore.kernel.org/[email protected]
- Save only the arguments into the ring buffer and not the ftrace_regs as the ftrace_regs should be opaque from generic code.
- Have the function tracer event be dynamic so that it can hold an array of arguments after the entry.
kernel/trace/trace.c | 12 ++++++-- kernel/trace/trace.h | 3 +- kernel/trace/trace_entries.h | 5 ++-- kernel/trace/trace_functions.c | 46 +++++++++++++++++++++++++++---- kernel/trace/trace_irqsoff.c | 4 +-- kernel/trace/trace_output.c | 18 ++++++++++-- kernel/trace/trace_sched_wakeup.c | 4 +-- 7 files changed, 75 insertions(+), 17 deletions(-)
kernel/trace/trace.c +10 -2
@@ -2897,13 +2897,16 @@ trace_buffer_unlock_commit_nostack(struct trace_buffer *buffer,void trace_function (struct trace_array *tr,unsigned long ip,unsigned long - parent_ip,unsigned int trace_ctx) + parent_ip,unsigned int trace_ctx,struct ftrace_regs *fregs) {struct trace_buffer *buffer = tr->array_buffer.buffer;struct ring_buffer_event *event;struct ftrace_entry *entry; +int size =sizeof (*entry); - event =__trace_buffer_lock_reserve (buffer,TRACE_FN ,sizeof (*entry), + size +=FTRACE_REGS_MAX_ARGS * !!fregs *sizeof (long ); + + event =__trace_buffer_lock_reserve (buffer,TRACE_FN , size, trace_ctx);if (!event)return ;
@@ -2911,6 +2914,11 @@ trace_function(struct trace_array *tr, unsigned long ip, unsigned long entry->ip = ip; entry->parent_ip = parent_ip; +if (fregs) { +for (int i =0 ; i <FTRACE_REGS_MAX_ARGS ; i++) + entry->args[i] =ftrace_regs_get_argument (fregs, i); + } +if (static_branch_unlikely (&trace_function_exports_enabled))ftrace_exports (event,TRACE_EXPORT_FUNCTION );__buffer_unlock_commit (buffer, event);
kernel/trace/trace.h +2 -1
@@ -685,7 +685,8 @@ unsigned long trace_total_entries(struct trace_array *tr);void trace_function (struct trace_array *tr,unsigned long ip,unsigned long parent_ip, -unsigned int trace_ctx); +unsigned int trace_ctx, +struct ftrace_regs *regs); void trace_graph_function(struct trace_array *tr, unsigned long ip, unsigned long parent_ip,
kernel/trace/trace_entries.h +3 -2
@@ -61,8 +61,9 @@ FTRACE_ENTRY_REG(function, ftrace_entry,TRACE_FN ,F_STRUCT ( - __field_fn( unsigned long, ip ) - __field_fn( unsigned long, parent_ip ) + __field_fn( unsigned long, ip ) + __field_fn( unsigned long, parent_ip ) + __dynamic_array( unsigned long, args ) ), F_printk(" %ps <-- %ps",
kernel/trace/trace_functions.c +41 -5
96 lines not on this page · raw message
kernel/trace/trace_irqsoff.c +2 -2
@@ -150,7 +150,7 @@ irqsoff_tracer_call(unsigned long ip, unsigned long parent_ip, trace_ctx =tracing_gen_ctx_flags (flags); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx, fregs);atomic_dec (&data->disabled); }
@@ -280,7 +280,7 @@ __trace_function(struct trace_array *tr,if (is_graph (tr))trace_graph_function (tr, ip, parent_ip, trace_ctx);else - trace_function(tr, ip, parent_ip, trace_ctx); + trace_function(tr, ip, parent_ip, trace_ctx, NULL); }#else
kernel/trace/trace_output.c +15 -3
@@ -1079,12 +1079,15 @@ enum print_line_t trace_nop_print(struct trace_iterator *iter, int flags, }static void print_fn_trace (struct trace_seq *s,unsigned long ip, -unsigned long parent_ip,long delta,int flags) +unsigned long parent_ip,long delta, +unsigned long *args,int flags) { ip += delta; parent_ip += delta;seq_print_ip_sym (s, ip, flags); +if (args) +print_function_args (s, args, ip);if ((flags &TRACE_ITER_PRINT_PARENT ) && parent_ip) {trace_seq_puts (s," <-" );
@@ -1098,10 +1101,19 @@ static enum print_line_t trace_fn_trace(struct trace_iterator *iter, int flags, {struct ftrace_entry *field;struct trace_seq *s = &iter->seq; +unsigned long *args; +int args_size;trace_assign_type (field, iter->ent); -print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, flags); + args_size = iter->ent_size - offsetof(struct ftrace_entry , args); +if (args_size >=FTRACE_REGS_MAX_ARGS *sizeof (long )) + args = field->args; +else + args =NULL ; + +print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, + args, flags);trace_seq_putc (s,'\n' );return trace_handle_return (s);
@@ -1774,7 +1786,7 @@ trace_func_repeats_print(struct trace_iterator *iter, int flags,trace_assign_type (field, iter->ent); -print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, flags); +print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta,NULL , flags);trace_seq_printf (s," (repeats: %u, last_ts:" , field->count);trace_print_time (s, iter, iter->ts -FUNC_REPEATS_GET_DELTA_TS (field));
kernel/trace/trace_sched_wakeup.c +2 -2
@@ -226,7 +226,7 @@ wakeup_tracer_call(unsigned long ip, unsigned long parent_ip,return ;local_irq_save (flags); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx, fregs);local_irq_restore (flags);atomic_dec (&data->disabled);
@@ -311,7 +311,7 @@ __trace_function(struct trace_array *tr,if (is_graph (tr))trace_graph_function (tr, ip, parent_ip, trace_ctx);else -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL ); }static int wakeup_flag_changed (struct trace_array *tr,u32 mask,int set)
Signature · 1 line
2.45.2
Steven Rostedt
maintainerRe: [PATCH v2 1/4] ftrace: Add print_function_args()
In reply to Steven RostedtQuoted 53 lines · On Mon, 23 Dec 2024 15:13:48 -0500 Steven Rostedt <[email protected]> wrote:
kernel/trace/trace_output.c +53
-0+#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) +{ +const struct btf_param *param; +const struct btf_type *t; +const char *param_name; +char name[KSYM_NAME_LEN ]; +unsigned long arg; +struct btf *btf; +s32 tid, nr =0 ; +int i; + +trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out; + + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out; + + t =btf_find_func_proto (name, &btf); +if (IS_ERR_OR_NULL (t)) + goto out; + + param =btf_get_func_param (t, &nr); +if (!param) + goto out_put; + +for (i =0 ; i < nr; i++) { +/* This only prints what the arch allows (6 args by default) */ +if (i ==FTRACE_REGS_MAX_ARGS ) { +trace_seq_puts (s,"..." ); +break ; + } + + arg = args[i]; + + param_name =btf_name_by_offset (btf, param[i].name_off); +if (param_name) +trace_seq_printf (s,"%s=" , param_name); + t =btf_type_skip_modifiers (btf, param[i].type, &tid); + +switch (t ?BTF_INFO_KIND (t->info) :BTF_KIND_UNKN ) { +case BTF_KIND_UNKN : +trace_seq_putc (s,'?' ); +/* Still print unknown type values */ + fallthrough; +case BTF_KIND_PTR : +trace_seq_printf (s,"0x%lx" , arg);
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 748
Quoted 10 lines
kernel/trace/trace_output.c +10
-0+break ; +case BTF_KIND_INT : +trace_seq_printf (s,"%ld" , arg); +break ; +case BTF_KIND_ENUM : +trace_seq_printf (s,"%ld" , arg); +break ; +default : +/* This does not handle complex arguments */ +trace_seq_printf (s,"0x%lx (%s)" , arg,btf_type_str (t));
This will need to take into account the size of the type. STRUCTs show up, and most of the time they are simply the size of the architecture's word. But I hit this:
timespec64_add_safe(lhs=0x6c2 (STRUCT), rhs=0x39c4153f (STRUCT))
Where it definitely isn't correct, as we have:
struct timespec64 {
time64_t tv_sec; /* seconds */
long tv_nsec; /* nanoseconds */
};struct timespec64 timespec64_add_safe(const struct timespec64 lhs,
const struct timespec64 rhs)
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 760
Quoted 12 lines
kernel/trace/trace_output.c +12
-0+ break; + } +if (i < nr -1 ) +trace_seq_printf (s,", " ); + } +out_put: +btf_put (btf); +out: +trace_seq_printf (s,")" ); +} +#endif +
I updated to this (separating the args array from the BTF parameters):
@@ -684,6 +683,91 @@ int trace_print_lat_context(struct trace_iterator *iter)
return !trace_seq_has_overflowed(s);
}
+#ifdef CONFIG_FUNCTION_TRACE_ARGS
+void print_function_args(struct trace_seq *s, unsigned long *args,
+ unsigned long func)
+{
+ const struct btf_param *param;
+ const struct btf_type *t;
+ const char *param_name;
+ char name[KSYM_NAME_LEN];
+ unsigned long arg;
+ struct btf *btf;
+ s32 tid, nr = 0;
+ int a, p, x;
+
+ trace_seq_printf(s, "(");
+
+ if (!args)
+ goto out;
+ if (lookup_symbol_name(func, name))
+ goto out;
+
+ btf = bpf_get_btf_vmlinux();
+ if (IS_ERR_OR_NULL(btf))
+ goto out;
+
+ t = btf_find_func_proto(name, &btf);
+ if (IS_ERR_OR_NULL(t))
+ goto out;
+
+ param = btf_get_func_param(t, &nr);
+ if (!param)
+ goto out_put;
+
+ for (a = 0, p = 0; p < nr; a++, p++) {
+ if (p)
+ trace_seq_puts(s, ", ");
+
+ /* This only prints what the arch allows (6 args by default) */
+ if (a == FTRACE_REGS_MAX_ARGS) {
+ trace_seq_puts(s, "...");
+ break;
+ }
+
+ arg = args[a];
+
+ param_name = btf_name_by_offset(btf, param[p].name_off);
+ if (param_name)
+ trace_seq_printf(s, "%s=", param_name);
+ t = btf_type_skip_modifiers(btf, param[p].type, &tid);
+
+ switch (t ? BTF_INFO_KIND(t->info) : BTF_KIND_UNKN) {
+ case BTF_KIND_UNKN:
+ trace_seq_putc(s, '?');
+ /* Still print unknown type values */
+ fallthrough;
+ case BTF_KIND_PTR:
+ trace_seq_printf(s, "0x%lx", arg);
+ break;
+ case BTF_KIND_INT:
+ trace_seq_printf(s, "%ld", arg);
+ break;
+ case BTF_KIND_ENUM:
+ trace_seq_printf(s, "%ld", arg);
+ break;
+ default:
+ /* This does not handle complex arguments */
+ trace_seq_printf(s, "(%s)[0x%lx", btf_type_str(t), arg);
+ for (x = sizeof(long); x < t->size; x += sizeof(long)) {
+ trace_seq_putc(s, ':');
+ if (++a == FTRACE_REGS_MAX_ARGS) {
+ trace_seq_puts(s, "...]");
+ goto out_put;
+ }
+ trace_seq_printf(s, "0x%lx", args[a]);
+ }
+ trace_seq_putc(s, ']');
+ break;
+ }
+ }
+out_put:
+ btf_put(btf);
+out:
+ trace_seq_printf(s, ")");
+}
+#endif
+
/**
* ftrace_find_event - find a registered event
* @type: the type of event to look for
And now I get this as output:
less-912 [006] ...1. 240.085082: timespec64_add_safe(lhs=(STRUCT)[0xef:0x10c8f152], rhs=(STRUCT)[0x4:0x0]) <-__se_sys_poll
less-914 [006] ...1. 241.241416: timespec64_add_safe(lhs=(STRUCT)[0xf0:0x1a1af859], rhs=(STRUCT)[0x4:0x0]) <-__se_sys_poll
less-916 [006] ...1. 242.653586: timespec64_add_safe(lhs=(STRUCT)[0xf1:0x32ac56c9], rhs=(STRUCT)[0x4:0x0]) <-__se_sys_poll
wpa_supplicant-488 [004] ...1. 246.771910: timespec64_add_safe(lhs=(STRUCT)[0xf5:0x39ba39d3], rhs=(STRUCT)[0x9:0x3b9ac618]) <-__se_sys_pselect6
wpa_supplicant-488 [004] ...1. 256.783841: timespec64_add_safe(lhs=(STRUCT)[0xff:0x3a714d62], rhs=(STRUCT)[0xa:0x0]) <-__se_sys_pselect6
NetworkManager-485 [006] ...1. 257.008653: timespec64_add_safe(lhs=(STRUCT)[0x100:0xc3ce7d5], rhs=(STRUCT)[0x0:0x16358818]) <-__se_sys_ppoll
NetworkManager-485 [006] ...1. 257.008676: timespec64_add_safe(lhs=(STRUCT)[0x100:0xc3d477e], rhs=(STRUCT)[0x0:0x16352288]) <-__se_sys_ppoll
NetworkManager-485 [006] ...1. 257.383973: timespec64_add_safe(lhs=(STRUCT)[0x100:0x229be0b6], rhs=(STRUCT)[0x11:0x11c46e70]) <-__se_sys_ppollWhich looks much more reasonable.
-- Steve
Masami Hiramatsu (Google)
maintainerRe: [PATCH v2 1/4] ftrace: Add print_function_args()
In reply to Steven RostedtQuoted 62 lines · On Mon, 23 Dec 2024 15:13:48 -0500 Steven Rostedt <[email protected]> wrote:
From: Sven Schnelle <[email protected]>
Add a function to decode argument types with the help of BTF. Will be used to display arguments in the function and function graph tracer.
It can only handle simply arguments and up to
FTRACE_REGS_MAX_ARGSnumber of arguments. When it hits a max, it will print ", ...":page_to_skb(vi=0xffff8d53842dc980, rq=0xffff8d53843a0800, page=0xfffffc2e04337c00, offset=6160, len=64, truesize=1536, ...)And if it hits an argument that is not recognized, it will print the raw value and the type of argument it is:
make_vfsuid(idmap=0xffffffff87f99db8, fs_userns=0xffffffff87e543c0, kuid=0x0 (STRUCT)) __pti_set_user_pgtbl(pgdp=0xffff8d5384ab47f8, pgd=0x110e74067 (STRUCT))Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Sven Schnelle <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Changes since v1: https://lore.kernel.org/[email protected]- Added Config option
FUNCTION_TRACE_ARGSto this patch (unconditional if dependencies are met)- Changed the
print_function_args()function to take an array of unsigned long args and not theftrace_regspointer. Theftrace_regsshould be opaque from generic code.- Have the function print the name of an BTF type that is not supported.
- Added
FTRACE_REGS_MAX_ARGSas the number of arguments saved in the event and printed out.- Print "...," if the number of arguments goes past
FTRACE_REGS_MAX_ARGS.include/linux/ftrace_regs.h | 5 +++ kernel/trace/Kconfig | 6 +++ kernel/trace/trace_output.c | 78 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace_output.h | 9 +++++ 4 files changed, 98 insertions(+)include/linux/ftrace_regs.h +5
-0@@ -35,4 +35,9 @@ struct ftrace_regs;#endif /* HAVE_ARCH_FTRACE_REGS */ +/* This can be overridden by the architectures */ +#ifndef FTRACE_REGS_MAX_ARGS +# define FTRACE_REGS_MAX_ARGS 6 +#endif +#endif /* _LINUX_FTRACE_REGS_H */ kernel/trace/Kconfig +0
-0
kernel/trace/Kconfig config FUNCTION_GRAPH_RETADDR · line 269
Quoted 7 lines
kernel/trace/Kconfig +4
-0@@ -263,6 +263,12 @@ config FUNCTION_GRAPH_RETADDR the function is called. This feature is off by default, and you can enable it via the trace option funcgraph-retaddr. +config FUNCTION_TRACE_ARGS + bool + depends on HAVE_FUNCTION_ARG_ACCESS_API + depends on DEBUG_INFO_BTF
For using the BTF APIs, we also needs BPF_SYSCALL (DEBUG_INFO_BTF just compiles the BTF info into the kernel binary.)
Others looks good to me.
Thanks!
Quoted 126 lines
kernel/trace/Kconfig +2
-0+ default y + config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically" depends on FUNCTION_TRACERkernel/trace/trace_output.c +78
-0@@ -12,8 +12,11 @@#include <linux/sched/clock.h> #include <linux/sched/mm.h> #include <linux/idr.h> +#include <linux/btf.h> +#include <linux/bpf.h> #include "trace_output.h" +#include "trace_btf.h" /* must be a power of 2 */ #define EVENT_HASHSIZE 128@@ -680,6 +683,81 @@ int trace_print_lat_context(struct trace_iterator *iter)return !trace_seq_has_overflowed (s); } +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) +{ +const struct btf_param *param; +const struct btf_type *t; +const char *param_name; +char name[KSYM_NAME_LEN ]; +unsigned long arg; +struct btf *btf; +s32 tid, nr =0 ; +int i; + +trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out; + + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out; + + t =btf_find_func_proto (name, &btf); +if (IS_ERR_OR_NULL (t)) + goto out; + + param =btf_get_func_param (t, &nr); +if (!param) + goto out_put; + +for (i =0 ; i < nr; i++) { +/* This only prints what the arch allows (6 args by default) */ +if (i ==FTRACE_REGS_MAX_ARGS ) { +trace_seq_puts (s,"..." ); +break ; + } + + arg = args[i]; + + param_name =btf_name_by_offset (btf, param[i].name_off); +if (param_name) +trace_seq_printf (s,"%s=" , param_name); + t =btf_type_skip_modifiers (btf, param[i].type, &tid); + +switch (t ?BTF_INFO_KIND (t->info) :BTF_KIND_UNKN ) { +case BTF_KIND_UNKN : +trace_seq_putc (s,'?' ); +/* Still print unknown type values */ + fallthrough; +case BTF_KIND_PTR : +trace_seq_printf (s,"0x%lx" , arg); +break ; +case BTF_KIND_INT : +trace_seq_printf (s,"%ld" , arg); +break ; +case BTF_KIND_ENUM : +trace_seq_printf (s,"%ld" , arg); +break ; +default : +/* This does not handle complex arguments */ +trace_seq_printf (s,"0x%lx (%s)" , arg,btf_type_str (t)); +break ; + } +if (i < nr -1 ) +trace_seq_printf (s,", " ); + } +out_put: +btf_put (btf); +out: +trace_seq_printf (s,")" ); +} +#endif +/** * ftrace_find_event - find a registered event * @type: the type of event to look for kernel/trace/trace_output.h +9
-0@@ -41,5 +41,14 @@ extern struct rw_semaphore trace_event_sem;#define SEQ_PUT_HEX_FIELD (s, x) \ trace_seq_putmem_hex(s, &(x), sizeof(x)) +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func); +#else +static inline void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) { +trace_seq_puts (s,"()" ); +} +#endif #endif Signature · 1 line
2.45.2
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH v2 4/4] ftrace: Add arguments to function tracer
In reply to Steven RostedtQuoted 8 lines · On Mon, 23 Dec 2024 15:13:51 -0500 Steven Rostedt <[email protected]> wrote:
From: Sven Schnelle <[email protected]>
Wire up the code to print function arguments in the function tracer. This functionality can be enabled/disabled during runtime with options/func-args.
ping-689 [004] b.... 77.170220: dummy_xmit(skb = 0x82904800, dev = 0x882d0000) <-dev_hard_start_xmit
Looks good to me.
Quoted 293 lines
Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Sven Schnelle <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Changes since v1: https://lore.kernel.org/[email protected]- Save only the arguments into the ring buffer and not the
ftrace_regsas theftrace_regsshould be opaque from generic code.- Have the function tracer event be dynamic so that it can hold an array of arguments after the entry.
kernel/trace/trace.c | 12 ++++++-- kernel/trace/trace.h | 3 +- kernel/trace/trace_entries.h | 5 ++-- kernel/trace/trace_functions.c | 46 +++++++++++++++++++++++++++---- kernel/trace/trace_irqsoff.c | 4 +-- kernel/trace/trace_output.c | 18 ++++++++++-- kernel/trace/trace_sched_wakeup.c | 4 +-- 7 files changed, 75 insertions(+), 17 deletions(-)kernel/trace/trace.c +10
-2@@ -2897,13 +2897,16 @@ trace_buffer_unlock_commit_nostack(struct trace_buffer *buffer,void trace_function (struct trace_array *tr,unsigned long ip,unsigned long - parent_ip,unsigned int trace_ctx) + parent_ip,unsigned int trace_ctx,struct ftrace_regs *fregs) {struct trace_buffer *buffer = tr->array_buffer.buffer;struct ring_buffer_event *event;struct ftrace_entry *entry; +int size =sizeof (*entry); - event =__trace_buffer_lock_reserve (buffer,TRACE_FN ,sizeof (*entry), + size +=FTRACE_REGS_MAX_ARGS * !!fregs *sizeof (long ); + + event =__trace_buffer_lock_reserve (buffer,TRACE_FN , size, trace_ctx);if (!event)return ;@@ -2911,6 +2914,11 @@ trace_function(struct trace_array *tr, unsigned long ip, unsigned long entry->ip = ip; entry->parent_ip = parent_ip; +if (fregs) { +for (int i =0 ; i <FTRACE_REGS_MAX_ARGS ; i++) + entry->args[i] =ftrace_regs_get_argument (fregs, i); + } +if (static_branch_unlikely (&trace_function_exports_enabled))ftrace_exports (event,TRACE_EXPORT_FUNCTION );__buffer_unlock_commit (buffer, event);kernel/trace/trace.h +2
-1@@ -685,7 +685,8 @@ unsigned long trace_total_entries(struct trace_array *tr);void trace_function (struct trace_array *tr,unsigned long ip,unsigned long parent_ip, -unsigned int trace_ctx); +unsigned int trace_ctx, +struct ftrace_regs *regs); void trace_graph_function(struct trace_array *tr, unsigned long ip, unsigned long parent_ip,kernel/trace/trace_entries.h +3
-2@@ -61,8 +61,9 @@ FTRACE_ENTRY_REG(function, ftrace_entry,TRACE_FN ,F_STRUCT ( - __field_fn( unsigned long, ip ) - __field_fn( unsigned long, parent_ip ) + __field_fn( unsigned long, ip ) + __field_fn( unsigned long, parent_ip ) + __dynamic_array( unsigned long, args ) ), F_printk(" %ps <-- %ps",kernel/trace/trace_functions.c +41
-5@@ -25,6 +25,9 @@ static void function_trace_call(unsigned long ip, unsigned long parent_ip, struct ftrace_ops *op, struct ftrace_regs *fregs);static void +function_args_trace_call (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *op,struct ftrace_regs *fregs); +static void function_stack_trace_call (unsigned long ip,unsigned long parent_ip,struct ftrace_ops *op,struct ftrace_regs *fregs); static void@@ -42,9 +45,10 @@ enum {TRACE_FUNC_NO_OPTS =0x0 ,/* No flags set. */ TRACE_FUNC_OPT_STACK =0x1 ,TRACE_FUNC_OPT_NO_REPEATS =0x2 , +TRACE_FUNC_OPT_ARGS =0x4 ,/* Update this to next highest bit. */ - TRACE_FUNC_OPT_HIGHEST_BIT = 0x4 + TRACE_FUNC_OPT_HIGHEST_BIT = 0x8 };#define TRACE_FUNC_OPT_MASK (TRACE_FUNC_OPT_HIGHEST_BIT - 1)@@ -114,6 +118,8 @@ static ftrace_func_t select_trace_function(u32 flags_val)switch (flags_val &TRACE_FUNC_OPT_MASK ) {case TRACE_FUNC_NO_OPTS :return function_trace_call; +case TRACE_FUNC_OPT_ARGS : +return function_args_trace_call;case TRACE_FUNC_OPT_STACK :return function_stack_trace_call;case TRACE_FUNC_OPT_NO_REPEATS :@@ -220,7 +226,34 @@ function_trace_call(unsigned long ip, unsigned long parent_ip, data =this_cpu_ptr (tr->array_buffer.data);if (!atomic_read (&data->disabled)) -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL ); + +ftrace_test_recursion_unlock (bit); +} + +static void +function_args_trace_call (unsigned long ip,unsigned long parent_ip, +struct ftrace_ops *op,struct ftrace_regs *fregs) +{ +struct trace_array *tr = op->private; +struct trace_array_cpu *data; +unsigned int trace_ctx; +int bit; +int cpu; + +if (unlikely (!tr->function_enabled)) +return ; + + bit =ftrace_test_recursion_trylock (ip, parent_ip); +if (bit <0 ) +return ; + + trace_ctx =tracing_gen_ctx (); + + cpu =smp_processor_id (); + data =per_cpu_ptr (tr->array_buffer.data, cpu); +if (!atomic_read (&data->disabled)) +trace_function (tr, ip, parent_ip, trace_ctx, fregs);ftrace_test_recursion_unlock (bit); }@@ -270,7 +303,7 @@ function_stack_trace_call(unsigned long ip, unsigned long parent_ip,if (likely (disabled ==1 )) { trace_ctx =tracing_gen_ctx_flags (flags); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL );#ifdef CONFIG_UNWINDER_FRAME_POINTER if (ftrace_pids_enabled (op)) skip++;@@ -351,7 +384,7 @@ function_no_repeats_trace_call(unsigned long ip, unsigned long parent_ip, trace_ctx =tracing_gen_ctx_flags (flags);process_repeats (tr, ip, parent_ip, last_info, trace_ctx); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL ); out:ftrace_test_recursion_unlock (bit);@@ -391,7 +424,7 @@ function_stack_no_repeats_trace_call(unsigned long ip, unsigned long parent_ip, trace_ctx =tracing_gen_ctx_flags (flags);process_repeats (tr, ip, parent_ip, last_info, trace_ctx); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL );__trace_stack (tr, trace_ctx,STACK_SKIP ); }@@ -405,6 +438,9 @@ static struct tracer_opt func_opts[] = { { TRACER_OPT(func_stack_trace, TRACE_FUNC_OPT_STACK) }, #endif { TRACER_OPT(func-no-repeats, TRACE_FUNC_OPT_NO_REPEATS) }, +#ifdef CONFIG_FUNCTION_TRACE_ARGS + { TRACER_OPT(func-args, TRACE_FUNC_OPT_ARGS) }, +#endif { } /* Always set a last empty entry */ };kernel/trace/trace_irqsoff.c +2
-2@@ -150,7 +150,7 @@ irqsoff_tracer_call(unsigned long ip, unsigned long parent_ip, trace_ctx =tracing_gen_ctx_flags (flags); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx, fregs);atomic_dec (&data->disabled); }@@ -280,7 +280,7 @@ __trace_function(struct trace_array *tr,if (is_graph (tr))trace_graph_function (tr, ip, parent_ip, trace_ctx);else - trace_function(tr, ip, parent_ip, trace_ctx); + trace_function(tr, ip, parent_ip, trace_ctx, NULL); }#else kernel/trace/trace_output.c +15
-3@@ -1079,12 +1079,15 @@ enum print_line_t trace_nop_print(struct trace_iterator *iter, int flags, }static void print_fn_trace (struct trace_seq *s,unsigned long ip, -unsigned long parent_ip,long delta,int flags) +unsigned long parent_ip,long delta, +unsigned long *args,int flags) { ip += delta; parent_ip += delta;seq_print_ip_sym (s, ip, flags); +if (args) +print_function_args (s, args, ip);if ((flags &TRACE_ITER_PRINT_PARENT ) && parent_ip) {trace_seq_puts (s," <-" );@@ -1098,10 +1101,19 @@ static enum print_line_t trace_fn_trace(struct trace_iterator *iter, int flags, {struct ftrace_entry *field;struct trace_seq *s = &iter->seq; +unsigned long *args; +int args_size;trace_assign_type (field, iter->ent); -print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, flags); + args_size = iter->ent_size - offsetof(struct ftrace_entry , args); +if (args_size >=FTRACE_REGS_MAX_ARGS *sizeof (long )) + args = field->args; +else + args =NULL ; + +print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, + args, flags);trace_seq_putc (s,'\n' );return trace_handle_return (s);@@ -1774,7 +1786,7 @@ trace_func_repeats_print(struct trace_iterator *iter, int flags,trace_assign_type (field, iter->ent); -print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta, flags); +print_fn_trace (s, field->ip, field->parent_ip, iter->tr->text_delta,NULL , flags);trace_seq_printf (s," (repeats: %u, last_ts:" , field->count);trace_print_time (s, iter, iter->ts -FUNC_REPEATS_GET_DELTA_TS (field));kernel/trace/trace_sched_wakeup.c +2
-2@@ -226,7 +226,7 @@ wakeup_tracer_call(unsigned long ip, unsigned long parent_ip,return ;local_irq_save (flags); -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx, fregs);local_irq_restore (flags);atomic_dec (&data->disabled);@@ -311,7 +311,7 @@ __trace_function(struct trace_array *tr,if (is_graph (tr))trace_graph_function (tr, ip, parent_ip, trace_ctx);else -trace_function (tr, ip, parent_ip, trace_ctx); +trace_function (tr, ip, parent_ip, trace_ctx,NULL ); }static int wakeup_flag_changed (struct trace_array *tr,u32 mask,int set)Signature · 1 line
2.45.2
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH v2 3/4] ftrace: Have funcgraph-args take affect during tracing
In reply to Steven RostedtQuoted 10 lines · On Mon, 23 Dec 2024 15:13:50 -0500 Steven Rostedt <[email protected]> wrote:
From: Steven Rostedt <[email protected]>
Currently, when
function_graphis started, it looks at the option funcgraph-args, and if it is set, it will enable tracing of the arguments.But if tracing is already running, and the user enables funcgraph-args, it will have no effect. Instead, it should enable argument tracing when it is enabled, even if it means disabling the function graph tracing for a short time in order to do the transition.
Looks good to me.
Quoted 61 lines
Signed-off-by Steven Rostedt (Google) <[email protected]>--- kernel/trace/trace_functions_graph.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-)kernel/trace/trace_functions_graph.c +26
-1@@ -469,7 +469,7 @@ static int graph_trace_init(struct trace_array *tr)else tr->gops->retfunc = trace_graph_return; -/* Make gops functions are visible before we start tracing */ +/* Make gops functions visible before we start tracing */ smp_mb (); ret =register_ftrace_graph (tr->gops);@@ -480,6 +480,28 @@ static int graph_trace_init(struct trace_array *tr)return 0 ; } +static int ftrace_graph_trace_args (struct trace_array *tr,int set) +{ +trace_func_graph_ent_t entry; + +if (set) + entry = trace_graph_entry_args; +else + entry = trace_graph_entry; + +/* See if there's any changes */ +if (tr->gops->entryfunc == entry) +return 0 ; + +unregister_ftrace_graph (tr->gops); + + tr->gops->entryfunc = entry; + +/* Make gops functions visible before we start tracing */ +smp_mb (); +return register_ftrace_graph (tr->gops); +} +static void graph_trace_reset (struct trace_array *tr) {tracing_stop_cmdline_record ();@@ -1609,6 +1631,9 @@ func_graph_set_flag(struct trace_array *tr, u32 old_flags, u32 bit, int set)if (bit ==TRACE_GRAPH_GRAPH_TIME )ftrace_graph_graph_time_control (set); +if (bit ==TRACE_GRAPH_ARGS ) +return ftrace_graph_trace_args (tr, set); +return 0 ; }Signature · 1 line
2.45.2
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Masami Hiramatsu (Google)
maintainerRe: [PATCH v2 2/4] ftrace: Add support for function argument to graph tracer
In reply to Steven RostedtQuoted 96 lines · On Mon, 23 Dec 2024 15:13:49 -0500 Steven Rostedt <[email protected]> wrote:
From: Sven Schnelle <[email protected]>
Wire up the code to print function arguments in the function graph tracer. This functionality can be enabled/disabled during runtime with options/funcgraph-args.
Example usage:
6) | dummy_xmit [dummy](skb = 0x8887c100, dev = 0x872ca000) { 6) | consume_skb(skb = 0x8887c100) { 6) | skb_release_head_state(skb = 0x8887c100) { 6) 0.178 us | sock_wfree(skb = 0x8887c100) 6) 0.627 us | }Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Sven Schnelle <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Changes since v1: https://lore.kernel.org/[email protected]- Record just
FTRACE_REGS_MAX_ARGSinto the ring buffer and not the entireftrace_regsstructure, as that structure should be opaque from generic code.- Make the function graph entry event have a dynamic size, so that an array of arguments may be added in the ring buffer after it.
kernel/trace/Kconfig | 6 ++ kernel/trace/trace.h | 1 + kernel/trace/trace_entries.h | 7 +- kernel/trace/trace_functions_graph.c | 147 +++++++++++++++++++++------ 4 files changed, 125 insertions(+), 36 deletions(-)kernel/trace/Kconfig +6
-0@@ -268,6 +268,12 @@ config FUNCTION_TRACE_ARGS depends on HAVE_FUNCTION_ARG_ACCESS_API depends on DEBUG_INFO_BTF default y + help + If supported with function argument access API and BTF, then + the function tracer and function graph tracer will support printing + of function arguments. This feature is off by default, and can be + enabled via the trace option func-args (for the function tracer) and + funcgraph-args (for the function graph tracer) config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically"kernel/trace/trace.h +1
-0@@ -887,6 +887,7 @@ static __always_inline bool ftrace_hash_empty(struct ftrace_hash *hash)#define TRACE_GRAPH_PRINT_RETVAL 0x800#define TRACE_GRAPH_PRINT_RETVAL_HEX 0x1000#define TRACE_GRAPH_PRINT_RETADDR 0x2000 +#define TRACE_GRAPH_ARGS 0x4000#define TRACE_GRAPH_PRINT_FILL_SHIFT 28#define TRACE_GRAPH_PRINT_FILL_MASK (0x3 << TRACE_GRAPH_PRINT_FILL_SHIFT)kernel/trace/trace_entries.h +4
-3@@ -72,17 +72,18 @@ FTRACE_ENTRY_REG(function, ftrace_entry, );/* Function call entry */ -FTRACE_ENTRY_PACKED (funcgraph_entry, ftrace_graph_ent_entry, +FTRACE_ENTRY (funcgraph_entry, ftrace_graph_ent_entry,TRACE_GRAPH_ENT ,F_STRUCT ( __field_struct( struct ftrace_graph_ent, graph_ent ) __field_packed( unsigned long, graph_ent, func ) -__field_packed ( int, graph_ent, depth ) + __field_packed( unsigned long, graph_ent, depth ) + __dynamic_array(unsigned long, args ) ), -F_printk ("--> %ps (%d)" , (void *)__entry->func, __entry->depth) +F_printk ("--> %ps (%lu)" , (void *)__entry->func, __entry->depth) );#ifdef CONFIG_FUNCTION_GRAPH_RETADDR kernel/trace/trace_functions_graph.c +0
-0
kernel/trace/trace_functions_graph.c line 16
Quoted 5 lines
kernel/trace/trace_functions_graph.c +2
-0@@ -12,6 +12,8 @@#include <linux/interrupt.h> #include <linux/slab.h> #include <linux/fs.h> +#include <linux/btf.h> +#include <linux/bpf.h>
Do we need these headers? I think it is wrapped by print_function_args().
Quoted 144 lines
kernel/trace/trace_functions_graph.c +58
-14#include "trace.h" #include "trace_output.h" @@ -70,6 +72,10 @@ static struct tracer_opt trace_opts[] = { #ifdef CONFIG_FUNCTION_GRAPH_RETADDR /* Display function return address ? */ { TRACER_OPT(funcgraph-retaddr, TRACE_GRAPH_PRINT_RETADDR) }, +#endif +#ifdef CONFIG_FUNCTION_TRACE_ARGS + /* Display function arguments ? */ + { TRACER_OPT(funcgraph-args, TRACE_GRAPH_ARGS) }, #endif /* Include sleep time (scheduled out) between entry and return */ { TRACER_OPT(sleep-time, TRACE_GRAPH_SLEEP_TIME) },@@ -110,25 +116,41 @@ static void print_graph_duration(struct trace_array *tr, unsigned long long duration, struct trace_seq *s, u32 flags); -int __trace_graph_entry (struct trace_array *tr, -struct ftrace_graph_ent *trace, -unsigned int trace_ctx) +static int __graph_entry (struct trace_array *tr,struct ftrace_graph_ent *trace, +unsigned int trace_ctx,struct ftrace_regs *fregs) {struct ring_buffer_event *event;struct trace_buffer *buffer = tr->array_buffer.buffer;struct ftrace_graph_ent_entry *entry; +int size; - event =trace_buffer_lock_reserve (buffer,TRACE_GRAPH_ENT , -sizeof (*entry), trace_ctx); +/* If fregs is defined, add FTRACE_REGS_MAX_ARGS long size words */ + size =sizeof (*entry) + (FTRACE_REGS_MAX_ARGS * !!fregs *sizeof (long )); + + event =trace_buffer_lock_reserve (buffer,TRACE_GRAPH_ENT , size, trace_ctx);if (!event)return 0 ; - entry =ring_buffer_event_data (event); - entry->graph_ent = *trace; + + entry =ring_buffer_event_data (event); + entry->graph_ent = *trace; + +if (fregs) { +for (int i =0 ; i <FTRACE_REGS_MAX_ARGS ; i++) + entry->args[i] =ftrace_regs_get_argument (fregs, i); + } +trace_buffer_unlock_commit_nostack (buffer, event);return 1 ; } +int __trace_graph_entry (struct trace_array *tr, +struct ftrace_graph_ent *trace, +unsigned int trace_ctx) +{ +return __graph_entry (tr, trace, trace_ctx,NULL ); +} +#ifdef CONFIG_FUNCTION_GRAPH_RETADDR int __trace_graph_retaddr_entry(struct trace_array *tr, struct ftrace_graph_ent *trace,@@ -174,9 +196,9 @@ struct fgraph_times {unsigned long long sleeptime;/* may be optional! */ }; -int trace_graph_entry (struct ftrace_graph_ent *trace, -struct fgraph_ops *gops, -struct ftrace_regs *fregs) +static int graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops, +struct ftrace_regs *fregs) {unsigned long *task_var =fgraph_get_task_var (gops);struct trace_array *tr = gops->private;@@ -248,7 +270,7 @@ int trace_graph_entry(struct ftrace_graph_ent *trace, ret =__trace_graph_retaddr_entry (tr, trace, trace_ctx, retaddr); }else - ret =__trace_graph_entry (tr, trace, trace_ctx); + ret =__graph_entry (tr, trace, trace_ctx, fregs); }else { ret =0 ; }@@ -259,6 +281,20 @@ int trace_graph_entry(struct ftrace_graph_ent *trace,return ret; } +int trace_graph_entry (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops, +struct ftrace_regs *fregs) +{ +return graph_entry (trace, gops,NULL ); +} + +static int trace_graph_entry_args (struct ftrace_graph_ent *trace, +struct fgraph_ops *gops, +struct ftrace_regs *fregs) +{ +return graph_entry (trace, gops, fregs); +} +static void __trace_graph_function (struct trace_array *tr,unsigned long ip,unsigned int trace_ctx)@@ -423,7 +459,10 @@ static int graph_trace_init(struct trace_array *tr) {int ret; - tr->gops->entryfunc = trace_graph_entry; +if (tracer_flags_is_set (TRACE_GRAPH_ARGS )) + tr->gops->entryfunc = trace_graph_entry_args; +else + tr->gops->entryfunc = trace_graph_entry;if (tracing_thresh) tr->gops->retfunc = trace_graph_thresh_return;@@ -780,7 +819,7 @@ static void print_graph_retaddr(struct trace_seq *s, struct fgraph_retaddr_ent_estatic void print_graph_retval (struct trace_seq *s,struct ftrace_graph_ent_entry *entry,struct ftrace_graph_ret *graph_ret,void *func, -u32 opt_flags,u32 trace_flags) +u32 opt_flags,u32 trace_flags,int args_size) {unsigned long err_code =0 ;unsigned long retval =0 ;@@ -814,7 +853,14 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entrif (entry->ent.type !=TRACE_GRAPH_RETADDR_ENT ) print_retaddr = false; -trace_seq_printf (s,"%ps();" , func); +trace_seq_printf (s,"%ps" , func); + +if (args_size >=FTRACE_REGS_MAX_ARGS *sizeof (long )) { +print_function_args (s, entry->args, (unsigned long )func); +trace_seq_putc (s,';' ); + }else +trace_seq_puts (s,"();" ); +if (print_retval || print_retaddr)trace_seq_puts (s," /*" );else
kernel/trace/trace_functions_graph.c static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr · line 885
Quoted 5 lines
kernel/trace/trace_functions_graph.c +1
-1@@ -836,12 +882,13 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr }if (!entry || print_retval || print_retaddr) -trace_seq_puts (s," */\n" ); +trace_seq_puts (s," */" );
Do we need this change?
Thank you,
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Quoted 87 lines · Steven Rostedt
These patches add support for printing function arguments in ftrace.
Example usage:
function tracer:
~# cd
/sys/kernel/tracing/
~# echoicmp_rcv>set_ftrace_filter
~# echo function >current_tracer
~# echo 1 >options/func-args
~# ping -c 10 127.0.0.1
[..]
~# cat trace
[..]ping-1277 [030] ..s1. 39.120939: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 39.120946: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 40.179724: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 40.179730: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 41.219700: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 41.219706: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 42.259717: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 42.259725: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 43.299735: icmp_rcv(skb=0xa0ecab00) <-ip_protocol_deliver_rcu ping-1277 [030] ..s1. 43.299742: icmp_rcv(skb=0xa0ecac00) <-ip_protocol_deliver_rcufunction graph:
~# cd
/sys/kernel/tracing
~# echoicmp_rcv>set_graph_function
~# echofunction_graph>current_tracer
~# echo 1 >options/funcgraph-args~# ping -c 1 127.0.0.1
~# cat trace
30) | icmp_rcv(skb=0xa0ecab00) { 30) | __skb_checksum_complete(skb=0xa0ecab00) { 30) | skb_checksum(skb=0xa0ecab00, offset=0, len=64, csum=0) { 30) | __skb_checksum(skb=0xa0ecab00, offset=0, len=64, csum=0, ops=0x232e0327a88) { 30) 0.418 us | csum_partial(buff=0xa0d20924, len=64, sum=0) 30) 0.985 us | } 30) 1.463 us | } 30) 2.039 us | }[..]
This was last posted by Sven Schnelle here:
https://lore.kernel.org/all/[email protected]/
As Sven hasn't worked on it since, I decided to continue to push it through. I'm keeping Sven as original author and added myself as "Co-developed-by".
The main changes are:
- Made the kconfig option unconditional if all the dependencies are set.
- Not save
ftrace_regsin the ring buffer, as that is an abstract descriptor defined by the architectures and should remain opaque from generic code. Instead, the args are read at the time they are recorded (with theftrace_regspassed to the callback function), and saved into the ring buffer. Then the print function only takes an array of elements.This could allow archs to retrieve arguments that are on the stack where as, post processing
ftrace_regscould cause undesirable results.- Made the function and function graph entry events dynamically sized to allow the arguments to be appended to the event in the ring buffer. The print function only looks to see if the event saved in the ring buffer is big enough to hold all the arguments defined by the new
FTRACE_REGS_MAX_ARGSmacro and if so, it will assume there are arguments there and print them. This also means user space will not break on reading these events as arguments will simply be ignored.- The printing of the arguments has some more data when things are not processed by BPF. Any unsupported argument will have the type printed out in the ring buffer.
- Also removed the spaces around the '=' as that's more in line to how trace events show their fields.
- One new patch I added to convert function graph tracing over to using args as soon as the user sets the option even if function graph tracing is enabled. Function tracer did this already by default.
Quoted 7 lines
Steven Rostedt (1):
ftrace: Have funcgraph-args take affect during tracingSven Schnelle (3):
ftrace: Addprint_function_args()
ftrace: Add support for function argument to graph tracer
ftrace: Add arguments to function tracer
hi,
what branch is this based on? can't find any that would apply patch#2 without conflict.
thanks,
jirka
Quoted 15 lines
----
include/linux/ftrace_regs.h | 5 + kernel/trace/Kconfig | 12 +++ kernel/trace/trace.c | 12 ++- kernel/trace/trace.h | 4 +- kernel/trace/trace_entries.h | 12 ++- kernel/trace/trace_functions.c | 46 ++++++++- kernel/trace/trace_functions_graph.c | 174 ++++++++++++++++++++++++++++------- kernel/trace/trace_irqsoff.c | 4 +- kernel/trace/trace_output.c | 96 ++++++++++++++++++- kernel/trace/trace_output.h | 9 ++ kernel/trace/trace_sched_wakeup.c | 4 +- 11 files changed, 324 insertions(+), 54 deletions(-)
Steven Rostedt
maintainerQuoted 3 lines · On Wed, 25 Dec 2024 22:40:53 +0100 Jiri Olsa <[email protected]> wrote:
hi,
what branch is this based on? can't find any that would apply patch#2 without conflict.
Sorry, I should have mentioned in the cover letter, that this is based on v6.13-rc4 with Masami's fprobe patchset applied:
https://lore.kernel.org/linux-trace-kernel/173379652547.973433.2311391879173461183.stgit@devnote2/
-- Steve
Steven Rostedt
maintainerRe: [PATCH v2 1/4] ftrace: Add print_function_args()
In reply to Masami Hiramatsu (Google)Quoted 10 lines · On Wed, 25 Dec 2024 22:15:55 +0900 Masami Hiramatsu (Google)…
kernel/trace/Kconfig config FUNCTION_GRAPH_RETADDR · line 269
Quoted 5 lines
kernel/trace/Kconfig +4
-0+config FUNCTION_TRACE_ARGS + bool + depends on HAVE_FUNCTION_ARG_ACCESS_API + depends on DEBUG_INFO_BTFFor using the BTF APIs, we also needs
BPF_SYSCALL(DEBUG_INFO_BTFjust compiles the BTF info into the kernel binary.)Others looks good to me.
Hmm, I removed it due to this feedback:
https://lore.kernel.org/linux-trace-kernel/[email protected]/
Quoted 2 lines
Revewied-by Masami Hiramatsu (Google) <[email protected]>
Thanks!
-- Steve
Steven Rostedt
maintainerRe: [PATCH v2 2/4] ftrace: Add support for function argument to graph tracer
In reply to Masami Hiramatsu (Google)Quoted 4 lines · On Thu, 26 Dec 2024 00:41:52 +0900 Masami Hiramatsu (Google)…
Quoted 4 lines
kernel/trace/trace_functions_graph.c +0
-0
Quoted 8 lines
kernel/trace/trace_functions_graph.c line 16
Quoted 6 lines
kernel/trace/trace_functions_graph.c +2
-0@@ -12,6 +12,8 @@#include <linux/interrupt.h> #include <linux/slab.h> #include <linux/fs.h> +#include <linux/btf.h> +#include <linux/bpf.h> Do we need these headers? I think it is wrapped by
print_function_args().
Oh, probably not. This is just leftovers from the original patch.
Quoted 19 lines
Quoted 19 lines
kernel/trace/trace_functions_graph.c +8
-1@@ -814,7 +853,14 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entrif (entry->ent.type !=TRACE_GRAPH_RETADDR_ENT ) print_retaddr = false; -trace_seq_printf (s,"%ps();" , func); +trace_seq_printf (s,"%ps" , func); + +if (args_size >=FTRACE_REGS_MAX_ARGS *sizeof (long )) { +print_function_args (s, entry->args, (unsigned long )func); +trace_seq_putc (s,';' ); + }else +trace_seq_puts (s,"();" ); +if (print_retval || print_retaddr)trace_seq_puts (s," /*" );else @@ -836,12 +882,13 @@ static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr }
Quoted 5 lines
kernel/trace/trace_functions_graph.c static void print_graph_retval(struct trace_seq *s, struct ftrace_graph_ent_entr · line 885
Quoted 3 lines
kernel/trace/trace_functions_graph.c +1
-1if (!entry || print_retval || print_retaddr) -trace_seq_puts (s," */\n" ); +trace_seq_puts (s," */" );Do we need this change?
Hmm, maybe not. It may have been caused by added (and removed) debugging.
Thanks for the review.
-- Steve
Masami Hiramatsu (Google)
maintainerRe: [PATCH v2 1/4] ftrace: Add print_function_args()
In reply to a message not in the archiveQuoted 120 lines · On Wed, 8 Jan 2025 10:30:08 +0800 Donglin Peng <[email protected]> wrote:
Quoted 118 lines · Steven Rostedt <[email protected]>于2024年12月24日 周二04:14写道:
From: Sven Schnelle <[email protected]>
Add a function to decode argument types with the help of BTF. Will be used to display arguments in the function and function graph tracer.
It can only handle simply arguments and up to
FTRACE_REGS_MAX_ARGSnumber of arguments. When it hits a max, it will print ", ...":page_to_skb(vi=0xffff8d53842dc980, rq=0xffff8d53843a0800,page=0xfffffc2e04337c00, offset=6160, len=64, truesize=1536, ...)
And if it hits an argument that is not recognized, it will print the raw value and the type of argument it is:
make_vfsuid(idmap=0xffffffff87f99db8, fs_userns=0xffffffff87e543c0,kuid=0x0 (STRUCT))
__pti_set_user_pgtbl(pgdp=0xffff8d5384ab47f8, pgd=0x110e74067 (STRUCT))Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Sven Schnelle <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Changes since v1:
https://lore.kernel.org/[email protected]- Added Config option
FUNCTION_TRACE_ARGSto this patch (unconditional if dependencies are met)- Changed the
print_function_args()function to take an array of unsigned long args and not theftrace_regspointer. Theftrace_regsshould be opaque from generic code.- Have the function print the name of an BTF type that is not supported.
- Added
FTRACE_REGS_MAX_ARGSas the number of arguments saved in the event and printed out.- Print "...," if the number of arguments goes past
FTRACE_REGS_MAX_ARGS.include/linux/ftrace_regs.h | 5 +++ kernel/trace/Kconfig | 6 +++ kernel/trace/trace_output.c | 78 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace_output.h | 9 +++++ 4 files changed, 98 insertions(+)include/linux/ftrace_regs.h +5
-0@@ -35,4 +35,9 @@ struct ftrace_regs;#endif /* HAVE_ARCH_FTRACE_REGS */ +/* This can be overridden by the architectures */ +#ifndef FTRACE_REGS_MAX_ARGS +# define FTRACE_REGS_MAX_ARGS 6 +#endif +#endif /* _LINUX_FTRACE_REGS_H */ kernel/trace/Kconfig +0
-0@@ -263,6 +263,12 @@ config FUNCTION_GRAPH_RETADDR the function is called. This feature is off by default, and youcan
enable it via the trace option funcgraph-retaddr.kernel/trace/Kconfig +6
-0+config FUNCTION_TRACE_ARGS + bool + depends on HAVE_FUNCTION_ARG_ACCESS_API + depends on DEBUG_INFO_BTF + default y + config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically" depends on FUNCTION_TRACERkernel/trace/trace_output.c +22
-0@@ -12,8 +12,11 @@#include <linux/sched/clock.h> #include <linux/sched/mm.h> #include <linux/idr.h> +#include <linux/btf.h> +#include <linux/bpf.h> #include "trace_output.h" +#include "trace_btf.h" /* must be a power of 2 */ #define EVENT_HASHSIZE 128@@ -680,6 +683,81 @@ int trace_print_lat_context(struct trace_iterator *iter)return !trace_seq_has_overflowed (s); } +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) +{ +const struct btf_param *param; +const struct btf_type *t; +const char *param_name; +char name[KSYM_NAME_LEN ]; +unsigned long arg; +struct btf *btf; +s32 tid, nr =0 ; +int i; + +trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out;
Quoted 8 lines
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 708
Quoted 4 lines
kernel/trace/trace_output.c +4
-0+ + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out;There is no need to the retrieve the BTF of vmlinux, as
btf_find_func_protowill return the correct BTF via its second parameter.
Good catch! The second parameter of btf_find_func_proto() is output.
Thank you,
Quoted 86 lines
— donglin
Quoted 82 lines
kernel/trace/trace_output.c +11
-0+ + t =btf_find_func_proto (name, &btf); +if (IS_ERR_OR_NULL (t)) + goto out; + + param =btf_get_func_param (t, &nr); +if (!param) + goto out_put; + +for (i =0 ; i < nr; i++) { +/* This only prints what the arch allows (6 args by default) */
kernel/trace/trace_output.c +29
-0+if (i ==FTRACE_REGS_MAX_ARGS ) { +trace_seq_puts (s,"..." ); +break ; + } + + arg = args[i]; + + param_name =btf_name_by_offset (btf, param[i].name_off); +if (param_name) +trace_seq_printf (s,"%s=" , param_name); + t =btf_type_skip_modifiers (btf, param[i].type, &tid); + +switch (t ?BTF_INFO_KIND (t->info) :BTF_KIND_UNKN ) { +case BTF_KIND_UNKN : +trace_seq_putc (s,'?' ); +/* Still print unknown type values */ + fallthrough; +case BTF_KIND_PTR : +trace_seq_printf (s,"0x%lx" , arg); +break ; +case BTF_KIND_INT : +trace_seq_printf (s,"%ld" , arg); +break ; +case BTF_KIND_ENUM : +trace_seq_printf (s,"%ld" , arg); +break ; +default : +/* This does not handle complex arguments */ + trace_seq_printf(s, "0x%lx (%s)", arg,
btf_type_str(t));kernel/trace/trace_output.c +12
-0+ break; + } +if (i < nr -1 ) +trace_seq_printf (s,", " ); + } +out_put: +btf_put (btf); +out: +trace_seq_printf (s,")" ); +} +#endif +/** * ftrace_find_event - find a registered event * @type: the type of event to look for kernel/trace/trace_output.h +9
-1@@ -41,5 +41,14 @@ extern struct rw_semaphore trace_event_sem;#define SEQ_PUT_HEX_FIELD (s, x) \ trace_seq_putmem_hex(s, &(x), sizeof(x)) +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func); +#else +static inline void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) { +trace_seq_puts (s,"()" ); +} +#endif #endif --2.45.2
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH v2 1/4] ftrace: Add print_function_args()
Quoted 132 lines · Masami Hiramatsu (Google)
Quoted 120 lines · On Wed, 8 Jan 2025 10:30:08 +0800 Donglin Peng <[email protected]> wrote:
Quoted 118 lines · Steven Rostedt <[email protected]>于2024年12月24日 周二04:14写道:
From: Sven Schnelle <[email protected]>
Add a function to decode argument types with the help of BTF. Will be used to display arguments in the function and function graph tracer.
It can only handle simply arguments and up to
FTRACE_REGS_MAX_ARGSnumber of arguments. When it hits a max, it will print ", ...":page_to_skb(vi=0xffff8d53842dc980, rq=0xffff8d53843a0800,page=0xfffffc2e04337c00, offset=6160, len=64, truesize=1536, ...)
And if it hits an argument that is not recognized, it will print the raw value and the type of argument it is:
make_vfsuid(idmap=0xffffffff87f99db8, fs_userns=0xffffffff87e543c0,kuid=0x0 (STRUCT))
__pti_set_user_pgtbl(pgdp=0xffff8d5384ab47f8, pgd=0x110e74067 (STRUCT))Co-developed-by Steven Rostedt (Google) <[email protected]>Signed-off-by Sven Schnelle <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>---
Changes since v1:
https://lore.kernel.org/[email protected]- Added Config option
FUNCTION_TRACE_ARGSto this patch (unconditional if dependencies are met)- Changed the
print_function_args()function to take an array of unsigned long args and not theftrace_regspointer. Theftrace_regsshould be opaque from generic code.- Have the function print the name of an BTF type that is not supported.
- Added
FTRACE_REGS_MAX_ARGSas the number of arguments saved in the event and printed out.- Print "...," if the number of arguments goes past
FTRACE_REGS_MAX_ARGS.include/linux/ftrace_regs.h | 5 +++ kernel/trace/Kconfig | 6 +++ kernel/trace/trace_output.c | 78 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace_output.h | 9 +++++ 4 files changed, 98 insertions(+)include/linux/ftrace_regs.h +5
-0@@ -35,4 +35,9 @@ struct ftrace_regs;#endif /* HAVE_ARCH_FTRACE_REGS */ +/* This can be overridden by the architectures */ +#ifndef FTRACE_REGS_MAX_ARGS +# define FTRACE_REGS_MAX_ARGS 6 +#endif +#endif /* _LINUX_FTRACE_REGS_H */ kernel/trace/Kconfig +0
-0@@ -263,6 +263,12 @@ config FUNCTION_GRAPH_RETADDR the function is called. This feature is off by default, and youcan
enable it via the trace option funcgraph-retaddr.kernel/trace/Kconfig +6
-0+config FUNCTION_TRACE_ARGS + bool + depends on HAVE_FUNCTION_ARG_ACCESS_API + depends on DEBUG_INFO_BTF + default y + config DYNAMIC_FTRACE bool "enable/disable function tracing dynamically" depends on FUNCTION_TRACERkernel/trace/trace_output.c +22
-0@@ -12,8 +12,11 @@#include <linux/sched/clock.h> #include <linux/sched/mm.h> #include <linux/idr.h> +#include <linux/btf.h> +#include <linux/bpf.h> #include "trace_output.h" +#include "trace_btf.h" /* must be a power of 2 */ #define EVENT_HASHSIZE 128@@ -680,6 +683,81 @@ int trace_print_lat_context(struct trace_iterator *iter)return !trace_seq_has_overflowed (s); } +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) +{ +const struct btf_param *param; +const struct btf_type *t; +const char *param_name; +char name[KSYM_NAME_LEN ]; +unsigned long arg; +struct btf *btf; +s32 tid, nr =0 ; +int i; + +trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out;Quoted 8 lines
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 708
Quoted 4 lines
kernel/trace/trace_output.c +4
-0+ + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out;There is no need to the retrieve the BTF of vmlinux, as
btf_find_func_protowill return the correct BTF via its second parameter.
Quoted 2 lines
Good catch! The second parameter of
btf_find_func_proto()is output.
One thought here - with btf_find_func_proto(), we will try kernel BTF and then proceed to module BTF, iterating over all modules to find the function prototype. So where we are tracing module functions this could get expensive if such a function is frequently encountered, and it also opens up the risk that we end up using the wrong function prototype from the wrong module that just happens to match on function name.
So I wonder if we could use the function address to do a more guided lookup. Perhaps we could use kallsyms_lookup(), retrieving the (potential) module name. Then maybe modify the signature of btf_find_func_proto() to take an optional module name parameter to avoid iteration? None of this is strictly needed, but it may speed things up a bit and give us more accurate parameter info for those few cases with name clashes, so could be done as a follow-up if needed. Thanks!
Alan
Quoted 90 lines
Thank you,
Quoted 86 lines
— donglin
Quoted 82 lines
kernel/trace/trace_output.c +11
-0+ + t =btf_find_func_proto (name, &btf); +if (IS_ERR_OR_NULL (t)) + goto out; + + param =btf_get_func_param (t, &nr); +if (!param) + goto out_put; + +for (i =0 ; i < nr; i++) { +/* This only prints what the arch allows (6 args by default) */
kernel/trace/trace_output.c +29
-0+if (i ==FTRACE_REGS_MAX_ARGS ) { +trace_seq_puts (s,"..." ); +break ; + } + + arg = args[i]; + + param_name =btf_name_by_offset (btf, param[i].name_off); +if (param_name) +trace_seq_printf (s,"%s=" , param_name); + t =btf_type_skip_modifiers (btf, param[i].type, &tid); + +switch (t ?BTF_INFO_KIND (t->info) :BTF_KIND_UNKN ) { +case BTF_KIND_UNKN : +trace_seq_putc (s,'?' ); +/* Still print unknown type values */ + fallthrough; +case BTF_KIND_PTR : +trace_seq_printf (s,"0x%lx" , arg); +break ; +case BTF_KIND_INT : +trace_seq_printf (s,"%ld" , arg); +break ; +case BTF_KIND_ENUM : +trace_seq_printf (s,"%ld" , arg); +break ; +default : +/* This does not handle complex arguments */ + trace_seq_printf(s, "0x%lx (%s)", arg,
btf_type_str(t));kernel/trace/trace_output.c +12
-0+ break; + } +if (i < nr -1 ) +trace_seq_printf (s,", " ); + } +out_put: +btf_put (btf); +out: +trace_seq_printf (s,")" ); +} +#endif +/** * ftrace_find_event - find a registered event * @type: the type of event to look for kernel/trace/trace_output.h +9
-1@@ -41,5 +41,14 @@ extern struct rw_semaphore trace_event_sem;#define SEQ_PUT_HEX_FIELD (s, x) \ trace_seq_putmem_hex(s, &(x), sizeof(x)) +#ifdef CONFIG_FUNCTION_TRACE_ARGS +void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func); +#else +static inline void print_function_args (struct trace_seq *s,unsigned long *args, +unsigned long func) { +trace_seq_puts (s,"()" ); +} +#endif #endif --2.45.2
Steven Rostedt
maintainerRe: [PATCH v2 1/4] ftrace: Add print_function_args()
Quoted 25 lines · On Wed, 8 Jan 2025 11:19:11 +0000 Alan Maguire <[email protected]> wrote:
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
kernel/trace/trace_output.c +6
-0+trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out;Quoted 8 lines
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 708
Quoted 4 lines
kernel/trace/trace_output.c +4
-0+ + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out;There is no need to the retrieve the BTF of vmlinux, as
btf_find_func_protowill return the correct BTF via its second parameter.Quoted 2 lines
Good catch! The second parameter of
btf_find_func_proto()is output.One thought here - with
btf_find_func_proto(), we will try kernel BTF and then proceed to module BTF, iterating over all modules to find the function prototype. So where we are tracing module functions this could get expensive if such a function is frequently encountered, and it also opens up the risk that we end up using the wrong function prototype from the wrong module that just happens to match on function name.
Quoted 7 lines
So I wonder if we could use the function address to do a more guided lookup. Perhaps we could use
kallsyms_lookup(), retrieving the (potential) module name. Then maybe modify the signature ofbtf_find_func_proto()to take an optional module name parameter to avoid iteration? None of this is strictly needed, but it may speed things up a bit and give us more accurate parameter info for those few cases with name clashes, so could be done as a follow-up if needed. Thanks!
Well, every place this is called, we first get the function name from kallsyms. Perhaps I can modify the code to get the module name as well, and if it exists, we can pass that too?
-- Steve
Re: [PATCH v2 1/4] ftrace: Add print_function_args()
Quoted 36 lines · Steven Rostedt
Quoted 25 lines · On Wed, 8 Jan 2025 11:19:11 +0000 Alan Maguire <[email protected]> wrote:
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
kernel/trace/trace_output.c +6
-0+trace_seq_printf (s,"(" ); + +if (!args) + goto out; +if (lookup_symbol_name (func, name)) + goto out;Quoted 8 lines
kernel/trace/trace_output.c int trace_print_lat_context(struct trace_iterator *iter) · line 708
Quoted 4 lines
kernel/trace/trace_output.c +4
-0+ + btf =bpf_get_btf_vmlinux (); +if (IS_ERR_OR_NULL (btf)) + goto out;There is no need to the retrieve the BTF of vmlinux, as
btf_find_func_protowill return the correct BTF via its second parameter.Quoted 2 lines
Good catch! The second parameter of
btf_find_func_proto()is output.One thought here - with
btf_find_func_proto(), we will try kernel BTF and then proceed to module BTF, iterating over all modules to find the function prototype. So where we are tracing module functions this could get expensive if such a function is frequently encountered, and it also opens up the risk that we end up using the wrong function prototype from the wrong module that just happens to match on function name.Quoted 7 lines
So I wonder if we could use the function address to do a more guided lookup. Perhaps we could use
kallsyms_lookup(), retrieving the (potential) module name. Then maybe modify the signature ofbtf_find_func_proto()to take an optional module name parameter to avoid iteration? None of this is strictly needed, but it may speed things up a bit and give us more accurate parameter info for those few cases with name clashes, so could be done as a follow-up if needed. Thanks!
Quoted 4 lines
Well, every place this is called, we first get the function name from kallsyms. Perhaps I can modify the code to get the module name as well, and if it exists, we can pass that too?
yep, that would be great! One thing we should probably think about for the future is ways to stash module/BTF ids such that we don't need lookup each time, but I can't see any obvious way to do that currently.
Quoted 1 line
-- Steve