tracing: Updates for 6.3
Subsystems (6)
Steven Rostedt
maintainercover tracing: Updates for 6.3
git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace.git
trace/for-next
Head SHA1: dc513fd5321d4c15ac1f820c224b2e5220b6e14f
Jia-Ju Bai (1):
tracing: Add NULL checks for buffer in ring_buffer_free_read_page()
Mark Rutland (1):
ftrace: Add sample with custom ops
Steven Rostedt (Google) (10):
tracing: Add a way to filter function addresses to function names tracing/selftests: Add test for event filtering on function name
tracing: Simplify calculating entry size using struct_size()
tracing: Allow stacktraces to be saved as histogram variables
tracing: Allow synthetic events to pass around stacktraces tracing/histogram: Add stacktrace type
tracing/histogram: Document variable stacktrace tracing/histogram: Add simple tests for stacktrace usage of synthetic events perf/tracing: Use stage6 of tracing to not duplicate macros bpf/tracing: Use stage6 of tracing to not duplicate macros
----
Documentation/trace/events.rst | 12 + Documentation/trace/histogram.rst | 129 +++++++++++ include/trace/bpf_probe.h | 45 +--- include/trace/perf.h | 46 +--- include/trace/stages/stage6_event_callback.h | 3 + kernel/trace/ring_buffer.c | 7 +- kernel/trace/trace.c | 2 +- kernel/trace/trace.h | 4 + kernel/trace/trace_events.c | 2 +- kernel/trace/trace_events_filter.c | 93 +++++++- kernel/trace/trace_events_hist.c | 63 ++++-- kernel/trace/trace_events_synth.c | 83 ++++++- kernel/trace/trace_synth.h | 1 + samples/Kconfig | 7 + samples/Makefile | 1 + samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-ops.c | 252 +++++++++++++++++++++ .../ftrace/test.d/filter/event-filter-function.tc | 58 +++++ .../inter-event/trigger-synthetic-event-stack.tc | 24 ++ .../inter-event/trigger-synthetic-event-syntax.tc | 6 + 20 files changed, 729 insertions(+), 110 deletions(-) create mode 100644 samples/ftrace/ftrace-ops.c create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc
Steven Rostedt
maintainerPATCH 1/12 tracing: Add a way to filter function addresses to function names
From: "Steven Rostedt (Google)" <[email protected]>
There's been several times where an event records a function address in its field and I needed to filter on that address for a specific function name. It required looking up the function in kallsyms, finding its size, and doing a compare of "field >= function_start && field < function_end".
But this would change from boot to boot and is unreliable in scripts. Also, it is useful to have this at boot up, where the addresses will not be known. For example, on the boot command line:
trace_trigger="initcall_finish.traceoff if func.function == acpi_init"
To implement this, add a ".function" prefix, that will check that the field is of size long, and the only operations allowed (so far) are "==" and "!=".
--- Documentation/trace/events.rst | 12 ++++ kernel/trace/trace_events.c | 2 +- kernel/trace/trace_events_filter.c | 93 +++++++++++++++++++++++++++++- 3 files changed, 105 insertions(+), 2 deletions(-)
Documentation/trace/events.rst +12 -0
@@ -207,6 +207,18 @@ field name:: As the kernel will have to know how to retrieve the memory that the pointer is at from user space. +You can convert any long type to a function address and search by function name:: + + call_site.function == security_prepare_creds + +The above will filter when the field "call_site" falls on the address within +"security_prepare_creds". That is, it will compare the value of "call_site" and +the filter will return true if it is greater than or equal to the start of +the function "security_prepare_creds" and less than the end of that function. + +The ".function" postfix can only be attached to values of size long, and can only +be compared with "==" or "!=". + 5.2 Setting filters -------------------
kernel/trace/trace_events.c +1 -1
@@ -2822,7 +2822,7 @@ static __init int setup_trace_triggers(char *str)if (!trigger)break ; bootup_triggers[i].event =strsep (&trigger,"." ); - bootup_triggers[i].trigger =strsep (&trigger,"." ); + bootup_triggers[i].trigger = trigger;if (!bootup_triggers[i].trigger)break ; }
kernel/trace/trace_events_filter.c +92 -1
143 lines not on this page · raw message
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 2/12 tracing/selftests: Add test for event filtering on function name
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
With the new filter logic of passing in the name of a function to match an instruction pointer (or the address of the function), add a test to make sure that it is functional.
This is also the first test to test plain filtering. The filtering has been tested via the trigger logic, which uses the same code, but there was nothing to test just the event filter, so this test is the first to add such a case.
--- .../test.d/filter/event-filter-function.tc | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc
tools/testing/selftests/ftrace/test.d/filter/event-filter-function.tc new file +58 -0
@@ -0,0 +1,58 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: event filter function - test event filtering on functions +# requires: set_event events/kmem/kmem_cache_free/filter +# flags: instance + +fail() { #msg + echo $1 + exit_fail +} + +echo "Test event filter function name" +echo 0 > tracing_on +echo 0 > events/enable +echo > trace +echo 'call_site.function == exit_mmap' > events/kmem/kmem_cache_free/filter +echo 1 > events/kmem/kmem_cache_free/enable +echo 1 > tracing_on +ls > /dev/null +echo 0 > events/kmem/kmem_cache_free/enable + +hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l` +misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l` + +if [ $hitcnt -eq 0 ]; then + exit_fail +fi + +if [ $misscnt -gt 0 ]; then + exit_fail +fi + +address=`grep ' exit_mmap$' /proc/kallsyms | cut -d' ' -f1` + +echo "Test event filter function address" +echo 0 > tracing_on +echo 0 > events/enable +echo > trace +echo "call_site.function == 0x$address" > events/kmem/kmem_cache_free/filter +echo 1 > events/kmem/kmem_cache_free/enable +echo 1 > tracing_on +sleep 1 +echo 0 > events/kmem/kmem_cache_free/enable + +hitcnt=`grep kmem_cache_free trace| grep exit_mmap | wc -l` +misscnt=`grep kmem_cache_free trace| grep -v exit_mmap | wc -l` + +if [ $hitcnt -eq 0 ]; then + exit_fail +fi + +if [ $misscnt -gt 0 ]; then + exit_fail +fi + +reset_events_filter + +exit 0
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 3/12 ftrace: Add sample with custom ops
In reply to Steven RostedtFrom: Mark Rutland <[email protected]>
When reworking core ftrace code or architectural ftrace code, it's often necessary to test/analyse/benchmark a number of ftrace_ops configurations. This patch adds a module which can be used to explore some of those configurations.
I'm using this to benchmark various options for changing the way trampolines and handling of ftrace_ops work on arm64, and ensuring other architectures aren't adversely affected.
For example, in a QEMU+KVM VM running on a 2GHz Xeon E5-2660 workstation, loading the module in various configurations produces:
| # insmod ftrace-ops.ko | ftrace_ops: registering: | relevant ops: 1 | tracee: tracee_relevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | irrelevant ops: 0 | tracee: tracee_irrelevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | saving registers: NO | assist recursion: NO | assist RCU: NO | ftrace_ops: Attempted 100000 calls to tracee_relevant [ftrace_ops] in 1681558ns (16ns / call)
| # insmod ftrace-ops.ko nr_ops_irrelevant=5 | ftrace_ops: registering: | relevant ops: 1 | tracee: tracee_relevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | irrelevant ops: 5 | tracee: tracee_irrelevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | saving registers: NO | assist recursion: NO | assist RCU: NO | ftrace_ops: Attempted 100000 calls to tracee_relevant [ftrace_ops] in 1693042ns (16ns / call)
| # insmod ftrace-ops.ko nr_ops_relevant=2 | ftrace_ops: registering: | relevant ops: 2 | tracee: tracee_relevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | irrelevant ops: 0 | tracee: tracee_irrelevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | saving registers: NO | assist recursion: NO | assist RCU: NO | ftrace_ops: Attempted 100000 calls to tracee_relevant [ftrace_ops] in 11965582ns (119ns / call)
| # insmod ftrace-ops.ko save_regs=true | ftrace_ops: registering: | relevant ops: 1 | tracee: tracee_relevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | irrelevant ops: 0 | tracee: tracee_irrelevant [ftrace_ops] | tracer: ops_func_nop [ftrace_ops] | saving registers: YES | assist recursion: NO | assist RCU: NO | ftrace_ops: Attempted 100000 calls to tracee_relevant [ftrace_ops] in 4459624ns (44ns / call)
--- samples/Kconfig | 7 + samples/Makefile | 1 + samples/ftrace/Makefile | 1 + samples/ftrace/ftrace-ops.c | 252 ++++++++++++++++++++++++++++++++++++ 4 files changed, 261 insertions(+) create mode 100644 samples/ftrace/ftrace-ops.c
samples/Kconfig +7 -0
@@ -46,6 +46,13 @@ config SAMPLE_FTRACE_DIRECT_MULTI that hooks to wake_up_process and schedule, and prints the function addresses. +config SAMPLE_FTRACE_OPS + tristate "Build custom ftrace ops example" + depends on FUNCTION_TRACER + help + This builds an ftrace ops example that hooks two functions and + measures the time taken to invoke one function a number of times. + config SAMPLE_TRACE_ARRAY tristate "Build sample module for kernel access to Ftrace instancess" depends on EVENT_TRACING && m
samples/Makefile +1 -0
@@ -24,6 +24,7 @@ obj-$(CONFIG_SAMPLE_TRACE_CUSTOM_EVENTS) += trace_events/ obj-$(CONFIG_SAMPLE_TRACE_PRINTK) += trace_printk/ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace/ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT_MULTI) += ftrace/ +obj-$(CONFIG_SAMPLE_FTRACE_OPS) += ftrace/ obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += ftrace/ subdir-$(CONFIG_SAMPLE_UHID) += uhid obj-$(CONFIG_VIDEO_PCI_SKELETON) += v4l/
samples/ftrace/Makefile +1 -0
@@ -5,6 +5,7 @@ obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-too.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT) += ftrace-direct-modify.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT_MULTI) += ftrace-direct-multi.o obj-$(CONFIG_SAMPLE_FTRACE_DIRECT_MULTI) += ftrace-direct-multi-modify.o +obj-$(CONFIG_SAMPLE_FTRACE_OPS) += ftrace-ops.o CFLAGS_sample-trace-array.o := -I$(src) obj-$(CONFIG_SAMPLE_TRACE_ARRAY) += sample-trace-array.o
samples/ftrace/ftrace-ops.c new file +252 -0
252 lines not on this page · raw message
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 4/12 tracing: Add NULL checks for buffer in ring_buffer_free_read_page()
In reply to Steven RostedtFrom: Jia-Ju Bai <[email protected]>
In a previous commit 7433632c9ff6, buffer, buffer->buffers and buffer->buffers[cpu] in ring_buffer_wake_waiters() can be NULL, and thus the related checks are added.
However, in the same call stack, these variables are also used in ring_buffer_free_read_page():
tracing_buffers_release()
ring_buffer_wake_waiters(iter->array_buffer->buffer)
cpu_buffer = buffer->buffers[cpu] -> Add checks by previous commit
ring_buffer_free_read_page(iter->array_buffer->buffer)
cpu_buffer = buffer->buffers[cpu] -> No checkThus, to avod possible null-pointer derefernces, the related checks should be added.
These results are reported by a static tool designed by myself.
--- kernel/trace/ring_buffer.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
kernel/trace/ring_buffer.c +6 -1
@@ -5626,11 +5626,16 @@ EXPORT_SYMBOL_GPL(ring_buffer_alloc_read_page);*/ void ring_buffer_free_read_page (struct trace_buffer *buffer,int cpu,void *data) { -struct ring_buffer_per_cpu *cpu_buffer = buffer->buffers[cpu]; +struct ring_buffer_per_cpu *cpu_buffer;struct buffer_data_page *bpage = data;struct page *page =virt_to_page (bpage);unsigned long flags; +if (!buffer || !buffer->buffers || !buffer->buffers[cpu]) +return ; + + cpu_buffer = buffer->buffers[cpu]; +/* If the page is still in use someplace else, we can't reuse it */ if (page_ref_count (page) >1 ) goto out;
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 5/12 tracing: Simplify calculating entry size using struct_size()
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
When tracing a dynamic string field for a synthetic event, the offset calculation for where to write the next event can use struct_size() to find what the current size of the structure is.
This simplifies the code and makes it less error prone.
--- kernel/trace/trace_events_synth.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
kernel/trace/trace_events_synth.c +1 -2
@@ -416,8 +416,7 @@ static unsigned int trace_string(struct synth_trace_event *entry,if (is_dynamic) {u32 data_offset; - data_offset = offsetof(typeof(*entry), fields); - data_offset += event->n_u64 *sizeof (u64); + data_offset =struct_size (entry, fields, event->n_u64); data_offset += data_size; len =kern_fetch_store_strlen ((unsigned long )str_val);
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 6/12 tracing: Allow stacktraces to be saved as histogram variables
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Allow to save stacktraces into a histogram variable. This will be used by synthetic events to allow a stacktrace from one event to be passed and displayed by another event.
The special keyword "stacktrace" is to be used to trigger a stack trace for the event that the histogram trigger is attached to.
echo 'hist:keys=pid:st=stacktrace" > events/sched/sched_waking/trigger
Currently nothing can get access to the "$st" variable above that contains the stack trace, but that will soon change.
--- kernel/trace/trace_events_hist.c | 52 ++++++++++++++++++++++++++------ 1 file changed, 42 insertions(+), 10 deletions(-)
kernel/trace/trace_events_hist.c +42 -10
@@ -1360,6 +1360,8 @@ static const char *hist_field_name(struct hist_field *field, field_name = field->name; }else if (field->flags &HIST_FIELD_FL_TIMESTAMP ) field_name ="common_timestamp" ; +else if (field->flags &HIST_FIELD_FL_STACKTRACE ) + field_name ="stacktrace" ;else if (field->flags &HIST_FIELD_FL_HITCOUNT ) field_name ="hitcount" ;
@@ -1980,6 +1982,10 @@ static struct hist_field *create_hist_field(struct hist_trigger_data *hist_data,if (flags &HIST_FIELD_FL_STACKTRACE ) { hist_field->fn_num =HIST_FIELD_FN_NOP ; + hist_field->size =HIST_STACKTRACE_SIZE ; + hist_field->type =kstrdup_const ("unsigned long[]" ,GFP_KERNEL ); +if (!hist_field->type) + goto free; goto out; }
@@ -2351,6 +2357,8 @@ parse_field(struct hist_trigger_data *hist_data, struct trace_event_file *file, hist_data->enable_timestamps = true;if (*flags &HIST_FIELD_FL_TIMESTAMP_USECS ) hist_data->attrs->ts_in_usecs = true; + }else if (strcmp (field_name,"stacktrace" ) ==0 ) { + *flags |=HIST_FIELD_FL_STACKTRACE ; }else if (strcmp (field_name,"common_cpu" ) ==0 ) *flags |=HIST_FIELD_FL_CPU ;else if (strcmp (field_name,"hitcount" ) ==0 )
@@ -3119,13 +3127,24 @@ static inline void __update_field_vars(struct tracing_map_elt *elt, var_val =hist_fn_call (val, elt, buffer, rbe, rec); var_idx = var->var.idx; -if (val->flags &HIST_FIELD_FL_STRING ) { +if (val->flags & (HIST_FIELD_FL_STRING | +HIST_FIELD_FL_STACKTRACE )) {char *str = elt_data->field_var_str[j++];char *val_str = (char *)(uintptr_t )var_val;unsigned int size; - size =min (val->size,STR_VAR_LEN_MAX ); -strscpy (str, val_str, size); +if (val->flags &HIST_FIELD_FL_STRING ) { + size =min (val->size,STR_VAR_LEN_MAX ); +strscpy (str, val_str, size); + }else { +int e; + + e =stack_trace_save ((void *)str, +HIST_STACKTRACE_DEPTH , +HIST_STACKTRACE_SKIP ); +if (e <HIST_STACKTRACE_DEPTH -1 ) + ((unsigned long *)str)[e] =0 ; + } var_val = (u64 )(uintptr_t )str; }tracing_map_set_var (elt, var_idx, var_val);
@@ -3824,7 +3843,8 @@ static void save_field_var(struct hist_trigger_data *hist_data, { hist_data->field_vars[hist_data->n_field_vars++] = field_var; -if (field_var->val->flags &HIST_FIELD_FL_STRING ) +/* Stack traces are saved in the string storage too */ +if (field_var->val->flags & (HIST_FIELD_FL_STRING |HIST_FIELD_FL_STACKTRACE )) hist_data->n_field_var_str++; }
@@ -4103,7 +4123,8 @@ static int action_create(struct hist_trigger_data *hist_data, } hist_data->save_vars[hist_data->n_save_vars++] = field_var; -if (field_var->val->flags &HIST_FIELD_FL_STRING ) +if (field_var->val->flags & + (HIST_FIELD_FL_STRING |HIST_FIELD_FL_STACKTRACE )) hist_data->n_save_var_str++;kfree (param); }
@@ -4351,7 +4372,8 @@ static int create_var_field(struct hist_trigger_data *hist_data,if (!ret && hist_data->fields[val_idx]->flags &HIST_FIELD_FL_EXECNAME )update_var_execname (hist_data->fields[val_idx]); -if (!ret && hist_data->fields[val_idx]->flags &HIST_FIELD_FL_STRING ) +if (!ret && hist_data->fields[val_idx]->flags & + (HIST_FIELD_FL_STRING |HIST_FIELD_FL_STACKTRACE )) hist_data->fields[val_idx]->var_str_idx = hist_data->n_var_str++;return ret;
@@ -5092,7 +5114,8 @@ static void hist_trigger_elt_update(struct hist_trigger_data *hist_data,if (hist_field->flags &HIST_FIELD_FL_VAR ) { var_idx = hist_field->var.idx; -if (hist_field->flags &HIST_FIELD_FL_STRING ) { +if (hist_field->flags & + (HIST_FIELD_FL_STRING |HIST_FIELD_FL_STACKTRACE )) {unsigned int str_start, var_str_idx, idx;char *str, *val_str;unsigned int size;
@@ -5105,9 +5128,18 @@ static void hist_trigger_elt_update(struct hist_trigger_data *hist_data, str = elt_data->field_var_str[idx]; val_str = (char *)(uintptr_t )hist_val; - size =min (hist_field->size,STR_VAR_LEN_MAX ); -strscpy (str, val_str, size); - +if (hist_field->flags &HIST_FIELD_FL_STRING ) { + size =min (hist_field->size,STR_VAR_LEN_MAX ); +strscpy (str, val_str, size); + }else { +int e; + + e =stack_trace_save ((void *)str, +HIST_STACKTRACE_DEPTH , +HIST_STACKTRACE_SKIP ); +if (e <HIST_STACKTRACE_DEPTH -1 ) + ((unsigned long *)str)[e] =0 ; + } hist_val = (u64 )(uintptr_t )str; }tracing_map_set_var (elt, var_idx, hist_val);
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 7/12 tracing: Allow synthetic events to pass around stacktraces
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Allow a stacktrace from one event to be displayed by the end event of a synthetic event. This is very useful when looking for the longest latency of a sleep or something blocked on I/O.
# cd /sys/kernel/tracing/ # echo 's:block_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events # echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 1||prev_state == 2' > events/sched/sched_switch/trigger # echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(block_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger
The above creates a "block_lat" synthetic event that take the stacktrace of when a task schedules out in either the interruptible or uninterruptible states, and on a new per process max $delta (the time it was scheduled out), will print the process id and the stacktrace.
# echo 1 > events/synthetic/block_lat/enable
# cat trace
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
kworker/u16:0-767 [006] d..4. 560.645045: block_lat: pid=767 delta=66 stack=STACK:
=> __schedule
=> schedule
=> pipe_read
=> vfs_read
=> ksys_read
=> do_syscall_64
=> 0x966000aa<idle>-0 [003] d..4. 561.132117: block_lat: pid=0 delta=413787 stack=STACK:
=> __schedule
=> schedule
=> schedule_hrtimeout_range_clock
=> do_sys_poll
=> __x64_sys_poll
=> do_syscall_64
=> 0x966000aa
<...>-153 [006] d..4. 562.068407: block_lat: pid=153 delta=54 stack=STACK:
=> __schedule
=> schedule
=> io_schedule
=> rq_qos_wait
=> wbt_wait
=> __rq_qos_throttle
=> blk_mq_submit_bio
=> submit_bio_noacct_nocheck
=> ext4_bio_write_page
=> mpage_submit_page
=> mpage_process_page_bufs
=> mpage_prepare_extent_to_map
=> ext4_do_writepages
=> ext4_writepages
=> do_writepages
=> __writeback_single_inode
--- kernel/trace/trace.h | 4 ++ kernel/trace/trace_events_hist.c | 7 ++- kernel/trace/trace_events_synth.c | 80 ++++++++++++++++++++++++++++++- kernel/trace/trace_synth.h | 1 + 4 files changed, 87 insertions(+), 5 deletions(-)
kernel/trace/trace.h +4 -0
@@ -113,6 +113,10 @@ enum trace_type {#define MEM_FAIL (condition, fmt, ...) \ DO_ONCE_LITE_IF(condition, pr_err, "ERROR: " fmt, ##__VA_ARGS__) +#define HIST_STACKTRACE_DEPTH 16 +#define HIST_STACKTRACE_SIZE (HIST_STACKTRACE_DEPTH * sizeof(unsigned long)) +#define HIST_STACKTRACE_SKIP 5 +/* * syscalls are special, and need special handling, this is why * they are not included in trace_entries.h
kernel/trace/trace_events_hist.c +3 -4
@@ -480,10 +480,6 @@ DEFINE_HIST_FIELD_FN(u8);#define for_each_hist_key_field (i, hist_data) \ for ((i) = (hist_data)->n_vals; (i) < (hist_data)->n_fields; (i)++) -#define HIST_STACKTRACE_DEPTH 16 -#define HIST_STACKTRACE_SIZE (HIST_STACKTRACE_DEPTH * sizeof(unsigned long)) -#define HIST_STACKTRACE_SKIP 5 -#define HITCOUNT_IDX 0#define HIST_KEY_SIZE_MAX (MAX_FILTER_STR_VAL + HIST_STACKTRACE_SIZE)
@@ -3869,6 +3865,9 @@ static int check_synth_field(struct synth_event *event, && field->is_dynamic)return 0 ; +if (strstr (hist_field->type,"long[" ) && field->is_stack) +return 0 ; +if (strcmp (field->type, hist_field->type) !=0 ) {if (field->size != hist_field->size || (!field->is_string && field->is_signed != hist_field->is_signed))
kernel/trace/trace_events_synth.c +79 -1
134 lines not on this page · raw message
kernel/trace/trace_synth.h +1 -0
@@ -18,6 +18,7 @@ struct synth_field {bool is_signed;bool is_string;bool is_dynamic; +bool is_stack; };struct synth_event {
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 8/12 tracing/histogram: Add stacktrace type
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Now that stacktraces can be part of synthetic events, allow a key to be typed as a stacktrace.
# cd /sys/kernel/tracing # echo 's:block_lat u64 delta; unsigned long stack[];' >> dynamic_events # echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 2' >> events/sched/sched_switch/trigger # echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,st2=$st:onmatch(sched.sched_switch).trace(block_lat,$delta,$st2)' >> events/sched/sched_switch/trigger # echo 'hist:keys=delta.buckets=100,stack.stacktrace:sort=delta' > events/synthetic/block_lat/trigger
# cat events/synthetic/block_lat/hist
# event histogram # # trigger info: hist:keys=delta.buckets=100,stacktrace:vals=hitcount:sort=delta.buckets=100:size=2048 [active] #
{ delta: ~ 0-99, stacktrace:
event_hist_trigger+0x464/0x480
event_triggers_call+0x52/0xe0
trace_event_buffer_commit+0x193/0x250
trace_event_raw_event_sched_switch+0xfc/0x150
__traceiter_sched_switch+0x41/0x60
__schedule+0x448/0x7b0
schedule_idle+0x26/0x40
cpu_startup_entry+0x19/0x20
start_secondary+0xed/0xf0
secondary_startup_64_no_verify+0xe0/0xeb
} hitcount: 6
{ delta: ~ 0-99, stacktrace:
event_hist_trigger+0x464/0x480
event_triggers_call+0x52/0xe0
trace_event_buffer_commit+0x193/0x250
trace_event_raw_event_sched_switch+0xfc/0x150
__traceiter_sched_switch+0x41/0x60
__schedule+0x448/0x7b0
schedule_idle+0x26/0x40
cpu_startup_entry+0x19/0x20
__pfx_kernel_init+0x0/0x10
arch_call_rest_init+0xa/0x24
start_kernel+0x964/0x98d
secondary_startup_64_no_verify+0xe0/0xeb
} hitcount: 3
{ delta: ~ 0-99, stacktrace:
event_hist_trigger+0x464/0x480
event_triggers_call+0x52/0xe0
trace_event_buffer_commit+0x193/0x250
trace_event_raw_event_sched_switch+0xfc/0x150
__traceiter_sched_switch+0x41/0x60
__schedule+0x448/0x7b0
schedule+0x5a/0xb0
worker_thread+0xaf/0x380
kthread+0xe9/0x110
ret_from_fork+0x2c/0x50
} hitcount: 1
{ delta: ~ 100-199, stacktrace:
event_hist_trigger+0x464/0x480
event_triggers_call+0x52/0xe0
trace_event_buffer_commit+0x193/0x250
trace_event_raw_event_sched_switch+0xfc/0x150
__traceiter_sched_switch+0x41/0x60
__schedule+0x448/0x7b0
schedule_idle+0x26/0x40
cpu_startup_entry+0x19/0x20
start_secondary+0xed/0xf0
secondary_startup_64_no_verify+0xe0/0xeb
} hitcount: 15
[..]
{ delta: ~ 8500-8599, stacktrace:
event_hist_trigger+0x464/0x480
event_triggers_call+0x52/0xe0
trace_event_buffer_commit+0x193/0x250
trace_event_raw_event_sched_switch+0xfc/0x150
__traceiter_sched_switch+0x41/0x60
__schedule+0x448/0x7b0
schedule_idle+0x26/0x40
cpu_startup_entry+0x19/0x20
start_secondary+0xed/0xf0
secondary_startup_64_no_verify+0xe0/0xeb
} hitcount: 1Totals:
Hits: 89
Entries: 11
Dropped: 0--- kernel/trace/trace_events_hist.c | 4 ++++ 1 file changed, 4 insertions(+)
kernel/trace/trace_events_hist.c +4 -0
@@ -1716,6 +1716,8 @@ static const char *get_hist_field_flags(struct hist_field *hist_field) flags_str ="percent" ; else if (hist_field->flags & HIST_FIELD_FL_GRAPH) flags_str ="graph" ; + else if (hist_field->flags & HIST_FIELD_FL_STACKTRACE) + flags_str ="stacktrace" ;return flags_str; }
@@ -2314,6 +2316,8 @@ parse_field(struct hist_trigger_data *hist_data, struct trace_event_file *file, *flags |=HIST_FIELD_FL_EXECNAME ; else if (strcmp(modifier, "syscall") == 0) *flags |= HIST_FIELD_FL_SYSCALL; + else if (strcmp(modifier, "stacktrace") == 0) + *flags |= HIST_FIELD_FL_STACKTRACE; else if (strcmp(modifier, "log2") == 0) *flags |= HIST_FIELD_FL_LOG2; else if (strcmp(modifier, "usecs") == 0)
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 9/12 tracing/histogram: Document variable stacktrace
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Add a little documentation (and a useful example) of how a stacktrace can be used within a histogram variable and synthetic event.
--- Documentation/trace/histogram.rst | 129 ++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+)
Documentation/trace/histogram.rst +129 -0
@@ -81,6 +81,7 @@ Documentation written by Tom Zanussi .usecs display a common_timestamp in microseconds .percent display a number of percentage value .graph display a bar-graph of a value + .stacktrace display as a stacktrace (must by a long[] type) ============= ================================================= Note that in general the semantics of a given field aren't
@@ -1786,6 +1787,8 @@ or assigned to a variable and referenced in a subsequent expression:: # echo 'hist:keys=next_pid:us_per_sec=1000000 ...' >> event/trigger # echo 'hist:keys=next_pid:timestamp_secs=common_timestamp/$us_per_sec ...' >> event/trigger +Variables can even hold stacktraces, which are useful with synthetic events. + 2.2.2 Synthetic Events ----------------------
@@ -1940,6 +1943,132 @@ the ".buckets" modifier and specify a size (in this case groups of 10). Entries: 16 Dropped: 0 +To save stacktraces, create a synthetic event with a field of type "unsigned long[]" +or even just "long[]". For example, to see how long a task is blocked in an +uninterruptible state: + + # cd /sys/kernel/tracing + # echo 's:block_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events + # echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 2' >> events/sched/sched_switch/trigger + # echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(block_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger + # echo 1 > events/synthetic/block_lat/enable + # cat trace + + # tracer: nop + # + # entries-in-buffer/entries-written: 2/2 #P:8 + # + # _-----=> irqs-off/BH-disabled + # / _----=> need-resched + # | / _---=> hardirq/softirq + # || / _--=> preempt-depth + # ||| / _-=> migrate-disable + # |||| / delay + # TASK-PID CPU# ||||| TIMESTAMP FUNCTION + # | | | ||||| | | + <idle>-0 [005] d..4. 521.164922: block_lat: pid=0 delta=8322 stack=STACK: + => __schedule+0x448/0x7b0 + => schedule+0x5a/0xb0 + => io_schedule+0x42/0x70 + => bit_wait_io+0xd/0x60 + => __wait_on_bit+0x4b/0x140 + => out_of_line_wait_on_bit+0x91/0xb0 + => jbd2_journal_commit_transaction+0x1679/0x1a70 + => kjournald2+0xa9/0x280 + => kthread+0xe9/0x110 + => ret_from_fork+0x2c/0x50 + + <...>-2 [004] d..4. 525.184257: block_lat: pid=2 delta=76 stack=STACK: + => __schedule+0x448/0x7b0 + => schedule+0x5a/0xb0 + => schedule_timeout+0x11a/0x150 + => wait_for_completion_killable+0x144/0x1f0 + => __kthread_create_on_node+0xe7/0x1e0 + => kthread_create_on_node+0x51/0x70 + => create_worker+0xcc/0x1a0 + => worker_thread+0x2ad/0x380 + => kthread+0xe9/0x110 + => ret_from_fork+0x2c/0x50 + +A synthetic event that has a stacktrace field may use it as a key in histogram: + + # echo 'hist:delta.buckets=100,stack.stacktrace:sort=delta' > events/synthetic/block_lat/trigger + # cat events/synthetic/block_lat/hist + + # event histogram + # + # trigger info: hist:keys=delta.buckets=100,stacktrace:vals=hitcount:sort=delta.buckets=100:size=2048 [active] + # + + { delta: ~ 0-99, stacktrace: + event_hist_trigger+0x464/0x480 + event_triggers_call+0x52/0xe0 + trace_event_buffer_commit+0x193/0x250 + trace_event_raw_event_sched_switch+0xfc/0x150 + __traceiter_sched_switch+0x41/0x60 + __schedule+0x448/0x7b0 + schedule_idle+0x26/0x40 + cpu_startup_entry+0x19/0x20 + start_secondary+0xed/0xf0 + secondary_startup_64_no_verify+0xe0/0xeb + } hitcount: 6 + { delta: ~ 0-99, stacktrace: + event_hist_trigger+0x464/0x480 + event_triggers_call+0x52/0xe0 + trace_event_buffer_commit+0x193/0x250 + trace_event_raw_event_sched_switch+0xfc/0x150 + __traceiter_sched_switch+0x41/0x60 + __schedule+0x448/0x7b0 + schedule_idle+0x26/0x40 + cpu_startup_entry+0x19/0x20 + __pfx_kernel_init+0x0/0x10 + arch_call_rest_init+0xa/0x24 + start_kernel+0x964/0x98d + secondary_startup_64_no_verify+0xe0/0xeb + } hitcount: 3 + { delta: ~ 0-99, stacktrace: + event_hist_trigger+0x464/0x480 + event_triggers_call+0x52/0xe0 + trace_event_buffer_commit+0x193/0x250 + trace_event_raw_event_sched_switch+0xfc/0x150 + __traceiter_sched_switch+0x41/0x60 + __schedule+0x448/0x7b0 + schedule+0x5a/0xb0 + worker_thread+0xaf/0x380 + kthread+0xe9/0x110 + ret_from_fork+0x2c/0x50 + } hitcount: 1 + { delta: ~ 100-199, stacktrace: + event_hist_trigger+0x464/0x480 + event_triggers_call+0x52/0xe0 + trace_event_buffer_commit+0x193/0x250 + trace_event_raw_event_sched_switch+0xfc/0x150 + __traceiter_sched_switch+0x41/0x60 + __schedule+0x448/0x7b0 + schedule_idle+0x26/0x40 + cpu_startup_entry+0x19/0x20 + start_secondary+0xed/0xf0 + secondary_startup_64_no_verify+0xe0/0xeb + } hitcount: 15 + [..] + { delta: ~ 8500-8599, stacktrace: + event_hist_trigger+0x464/0x480 + event_triggers_call+0x52/0xe0 + trace_event_buffer_commit+0x193/0x250 + trace_event_raw_event_sched_switch+0xfc/0x150 + __traceiter_sched_switch+0x41/0x60 + __schedule+0x448/0x7b0 + schedule_idle+0x26/0x40 + cpu_startup_entry+0x19/0x20 + start_secondary+0xed/0xf0 + secondary_startup_64_no_verify+0xe0/0xeb + } hitcount: 1 + + Totals: + Hits: 89 + Entries: 11 + Dropped: 0 + 2.2.3 Hist trigger 'handlers' and 'actions' -------------------------------------------
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 10/12 tracing/histogram: Add simple tests for stacktrace usage of synthetic events
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Update the selftests to include a test of passing a stacktrace between the events of a synthetic event.
--- kernel/trace/trace.c | 2 +- .../trigger-synthetic-event-stack.tc | 24 +++++++++++++++++++ .../trigger-synthetic-event-syntax.tc | 6 +++++ 3 files changed, 31 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc
kernel/trace/trace.c +1 -1
@@ -5757,7 +5757,7 @@ static const char readme_msg[] =#ifdef CONFIG_SYNTH_EVENTS " events/synthetic_events\t- Create/append/remove/show synthetic events\n" "\t Write into this file to define/undefine new synthetic events.\n" - "\t example: echo 'myevent u64 lat; char name[]' >> synthetic_events\n" + "\t example: echo 'myevent u64 lat; char name[]; long[] stack' >> synthetic_events\n"#endif #endif ;
tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-stack.tc new file +24 -0
@@ -0,0 +1,24 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: event trigger - test inter-event histogram trigger trace action with dynamic string param +# requires: set_event synthetic_events events/sched/sched_process_exec/hist "long[]' >> synthetic_events":README + +fail() { #msg + echo $1 + exit_fail +} + +echo "Test create synthetic event with stack" + + +echo 's:wake_lat pid_t pid; u64 delta; unsigned long[] stack;' > dynamic_events +echo 'hist:keys=next_pid:ts=common_timestamp.usecs,st=stacktrace if prev_state == 1||prev_state == 2' >> events/sched/sched_switch/trigger +echo 'hist:keys=prev_pid:delta=common_timestamp.usecs-$ts,s=$st:onmax($delta).trace(wake_lat,prev_pid,$delta,$s)' >> events/sched/sched_switch/trigger +echo 1 > events/synthetic/wake_lat/enable +sleep 1 + +if ! grep -q "=>.*sched" trace; then + fail "Failed to create synthetic event with stack" +fi + +exit 0
tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-syntax.tc +6 -0
@@ -70,6 +70,12 @@ grep "myevent[[:space:]]unsigned long var" synthetic_events echo "myevent char var[10]" > synthetic_events grep "myevent[[:space:]]char\[10\] var" synthetic_events +if grep -q 'long\[\]' README; then + # test stacktrace type + echo "myevent unsigned long[] var" > synthetic_events + grep "myevent[[:space:]]unsigned long\[\] var" synthetic_events +fi + do_reset exit 0
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 11/12 perf/tracing: Use stage6 of tracing to not duplicate macros
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The perf events are created by the same macro magic as tracefs trace events are. But to hook into perf, it has its own code. It duplicates many of the same macros as the tracefs macros and this is an issue because it misses bug fixes as well as any new enhancements that come with the other trace macros.
As the trace macros have been put into their own staging files, have perf take advantage of this and use the tracefs stage 6 macros that the "fast assign" portion of the trace event macro uses.
--- include/trace/perf.h | 46 +------------------- include/trace/stages/stage6_event_callback.h | 3 ++ 2 files changed, 4 insertions(+), 45 deletions(-)
include/trace/perf.h +1 -45
@@ -4,51 +4,7 @@#ifdef CONFIG_PERF_EVENTS -#undef __entry -#define __entry entry - -#undef __get_dynamic_array -#define __get_dynamic_array (field) \ - ((void *)__entry + (__entry->__data_loc_##field & 0xffff)) - -#undef __get_dynamic_array_len -#define __get_dynamic_array_len (field) \ - ((__entry->__data_loc_##field >> 16) & 0xffff) - -#undef __get_str -#define __get_str (field) ((char *)__get_dynamic_array(field)) - -#undef __get_bitmask -#define __get_bitmask (field) (char *)__get_dynamic_array(field) - -#undef __get_cpumask -#define __get_cpumask (field) (char *)__get_dynamic_array(field) - -#undef __get_sockaddr -#define __get_sockaddr (field) ((struct sockaddr *)__get_dynamic_array(field)) - -#undef __get_rel_dynamic_array -#define __get_rel_dynamic_array (field) \ - ((void *)__entry + \ - offsetof(typeof(*__entry), __rel_loc_##field) + \ - sizeof(__entry->__rel_loc_##field) + \ - (__entry->__rel_loc_##field & 0xffff)) - -#undef __get_rel_dynamic_array_len -#define __get_rel_dynamic_array_len (field) \ - ((__entry->__rel_loc_##field >> 16) & 0xffff) - -#undef __get_rel_str -#define __get_rel_str (field) ((char *)__get_rel_dynamic_array(field)) - -#undef __get_rel_bitmask -#define __get_rel_bitmask (field) (char *)__get_rel_dynamic_array(field) - -#undef __get_rel_cpumask -#define __get_rel_cpumask (field) (char *)__get_rel_dynamic_array(field) - -#undef __get_rel_sockaddr -#define __get_rel_sockaddr (field) ((struct sockaddr *)__get_rel_dynamic_array(field)) +#include "stages/stage6_event_callback.h" #undef __perf_count#define __perf_count (c) (__count = (c))
include/trace/stages/stage6_event_callback.h +3 -0
@@ -2,6 +2,9 @@/* Stage 6 definitions for creating trace events */ +/* Reuse some of the stage 3 macros */ +#include "stage3_trace_output.h" +#undef __entry#define __entry entry
Signature · 1 line
2.39.0
Steven Rostedt
maintainerPATCH 12/12 bpf/tracing: Use stage6 of tracing to not duplicate macros
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The bpf events are created by the same macro magic as tracefs trace events are. But to hook into bpf, it has its own code. It duplicates many of the same macros as the tracefs macros and this is an issue because it misses bug fixes as well as any new enhancements that come with the other trace macros.
As the trace macros have been put into their own staging files, have bpf take advantage of this and use the tracefs stage 6 macros that the "fast ssign" portion of the trace event macro uses.
--- include/trace/bpf_probe.h | 45 +-------------------------------------- 1 file changed, 1 insertion(+), 44 deletions(-)
include/trace/bpf_probe.h +1 -44
@@ -4,50 +4,7 @@#ifdef CONFIG_BPF_EVENTS -#undef __entry -#define __entry entry - -#undef __get_dynamic_array -#define __get_dynamic_array (field) \ - ((void *)__entry + (__entry->__data_loc_##field & 0xffff)) - -#undef __get_dynamic_array_len -#define __get_dynamic_array_len (field) \ - ((__entry->__data_loc_##field >> 16) & 0xffff) - -#undef __get_str -#define __get_str (field) ((char *)__get_dynamic_array(field)) - -#undef __get_bitmask -#define __get_bitmask (field) (char *)__get_dynamic_array(field) - -#undef __get_cpumask -#define __get_cpumask (field) (char *)__get_dynamic_array(field) - -#undef __get_sockaddr -#define __get_sockaddr (field) ((struct sockaddr *)__get_dynamic_array(field)) - -#undef __get_rel_dynamic_array -#define __get_rel_dynamic_array (field) \ - ((void *)(&__entry->__rel_loc_##field) + \ - sizeof(__entry->__rel_loc_##field) + \ - (__entry->__rel_loc_##field & 0xffff)) - -#undef __get_rel_dynamic_array_len -#define __get_rel_dynamic_array_len (field) \ - ((__entry->__rel_loc_##field >> 16) & 0xffff) - -#undef __get_rel_str -#define __get_rel_str (field) ((char *)__get_rel_dynamic_array(field)) - -#undef __get_rel_bitmask -#define __get_rel_bitmask (field) (char *)__get_rel_dynamic_array(field) - -#undef __get_rel_cpumask -#define __get_rel_cpumask (field) (char *)__get_rel_dynamic_array(field) - -#undef __get_rel_sockaddr -#define __get_rel_sockaddr (field) ((struct sockaddr *)__get_rel_dynamic_array(field)) +#include "stages/stage6_event_callback.h" #undef __perf_count#define __perf_count (c) (c)
Signature · 1 line
2.39.0