tracing/boot: Add new options for tracing specific period
Subsystems (3)
Masami Hiramatsu
maintainerv2 cover tracing/boot: Add new options for tracing specific period
Hi,
Here is the 2nd version of the series to improve the boot-time tracing to support kretprobe and tracing_on option. Previous version is here:
https://lkml.kernel.org/r/159887792384.1330989.5993224243767476896.stgit@devnote2
This version fixes a build error ([1/6]) and fix a space indent issue ([5/6]).
The combination of tracing_on and kretprobe allows us to trace events while a specific function call period. For example, the below bootconfig will make a function callgraph in the pci_proc_init() function at boot time.
ftrace {
tracing_on = 0 # off at start
tracer = function_graph
event.kprobes {
start_event {
probes = "pci_proc_init"
actions = "traceon"
}
end_event {
probes = "pci_proc_init%return"
actions = "traceoff"
}
}
}Here is the example output;
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
0) | pci_proc_init() {
0) | proc_mkdir() {
0) | proc_mkdir_data() {
0) | __proc_create() {
0) | _raw_read_lock() {
0) 0.179 us | preempt_count_add();
0) 0.203 us | do_raw_read_lock();
0) 1.210 us | }
0) | __xlate_proc_name() {
0) 0.449 us | pde_subdir_find();
0) 0.913 us | }
0) | _raw_read_unlock() {
0) 0.169 us | do_raw_read_unlock();
0) 0.175 us | preempt_count_sub();
0) 0.841 us | }
0) | kmem_cache_alloc() {
0) | fs_reclaim_acquire() {
0) 0.154 us | __need_fs_reclaim();
0) 0.240 us | fs_reclaim_acquire.part.0();
0) 0.889 us | }
0) | fs_reclaim_release() {
0) 0.174 us | __need_fs_reclaim();
0) 0.516 us | }
0) 0.157 us | should_failslab();
0) | rcu_read_lock_sched_held() {
0) | rcu_read_lock_held_common() {
0) 0.156 us | rcu_is_watching();
0) 0.158 us | rcu_lockdep_current_cpu_online();
0) 0.735 us | }
0) 1.054 us | }
0) 3.407 us | }
0) 0.168 us | __raw_spin_lock_init();
0) 7.575 us | }
0) | proc_register() {
0) | _raw_spin_lock_irqsave() {
0) 0.187 us | preempt_count_add();...
Thank you,
---
Masami Hiramatsu (6):
kprobes: tracing/kprobes: Fix to kill kprobes on initmem after boot tracing/boot: Add per-instance tracing_on option support
Documentation: tracing: Add tracing_on option to boot-time tracer tracing/kprobes: Support perf-style return probe
Documentation: tracing: Add %return suffix description
Documentation: tracing: boot: Add an example of tracing function-calls
Documentation/trace/boottime-trace.rst | 24 ++++++++++++++++++++++++ Documentation/trace/kprobetrace.rst | 2 ++ include/linux/kprobes.h | 5 +++++ init/main.c | 2 ++ kernel/kprobes.c | 22 ++++++++++++++++++++++ kernel/trace/trace_boot.c | 10 ++++++++++ kernel/trace/trace_kprobe.c | 21 ++++++++++++++++++++- 7 files changed, 85 insertions(+), 1 deletion(-)
--
Masami Hiramatsu (Linaro) <[email protected]>
Masami Hiramatsu
maintainerPATCH v2 1/6 kprobes: tracing/kprobes: Fix to kill kprobes on initmem after boot
Since kprobe_event= cmdline option allows user to put kprobes on the functions in initmem, kprobe has to make such probes gone after boot. Currently the probes on the init functions in modules will be handled by module callback, but the kernel init text isn't handled. Without this, kprobes may access non-exist text area to disable or remove it.
---
Changes in v2:
- Fix kprobe_free_init_mem() not depending on CONFIG_DEBUG_FS.
--- include/linux/kprobes.h | 5 +++++ init/main.c | 2 ++ kernel/kprobes.c | 22 ++++++++++++++++++++++ 3 files changed, 29 insertions(+)
include/linux/kprobes.h +5 -0
@@ -373,6 +373,8 @@ void unregister_kretprobes(struct kretprobe **rps, int num);void kprobe_flush_task (struct task_struct *tk);void recycle_rp_inst (struct kretprobe_instance *ri,struct hlist_head *head); +void kprobe_free_init_mem (void ); +int disable_kprobe (struct kprobe *kp);int enable_kprobe (struct kprobe *kp);
@@ -435,6 +437,9 @@ static inline void unregister_kretprobes(struct kretprobe **rps, int num)static inline void kprobe_flush_task (struct task_struct *tk) { } +static inline void kprobe_free_init_mem (void ) +{ +}static inline int disable_kprobe (struct kprobe *kp) {return -ENOSYS ;
init/main.c +2 -0
@@ -33,6 +33,7 @@#include <linux/nmi.h> #include <linux/percpu.h> #include <linux/kmod.h> +#include <linux/kprobes.h> #include <linux/vmalloc.h> #include <linux/kernel_stat.h> #include <linux/start_kernel.h>
@@ -1402,6 +1403,7 @@ static int __ref kernel_init(void *unused)kernel_init_freeable ();/* need to finish all async __init code before freeing the memory */ async_synchronize_full (); +kprobe_free_init_mem ();ftrace_free_init_mem ();free_initmem ();mark_readonly ();
kernel/kprobes.c +22 -0
@@ -2452,6 +2452,28 @@ static struct notifier_block kprobe_module_nb = {extern unsigned long __start_kprobe_blacklist[];extern unsigned long __stop_kprobe_blacklist[]; +void kprobe_free_init_mem (void ) +{ +void *start = (void *)(&__init_begin); +void *end = (void *)(&__init_end); +struct hlist_head *head; +struct kprobe *p; +int i; + +mutex_lock (&kprobe_mutex); + +/* Kill all kprobes on initmem */ +for (i =0 ; i <KPROBE_TABLE_SIZE ; i++) { + head = &kprobe_table[i]; +hlist_for_each_entry (p, head, hlist) { +if (start <= (void *)p->addr && (void *)p->addr < end) +kill_kprobe (p); + } + } + +mutex_unlock (&kprobe_mutex); +} +static int __init init_kprobes (void ) {int i, err =0 ;
Masami Hiramatsu
maintainerPATCH v2 2/6 tracing/boot: Add per-instance tracing_on option support
In reply to Masami HiramatsuAdd per-instance tracing_on option, which will be useful with traceon/traceoff event trigger actions. For example, if we disable tracing_on by default and set traceon and traceoff on a pair of events, we can trace functions between the pair of events.
--- kernel/trace/trace_boot.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
kernel/trace/trace_boot.c +10 -0
@@ -40,6 +40,16 @@ trace_boot_set_instance_options(struct trace_array *tr, struct xbc_node *node)pr_err ("Failed to set option: %s\n" , buf); } + p =xbc_node_find_value (node,"tracing_on" ,NULL ); +if (p && *p !='\0' ) { +if (kstrtoul (p,10 , &v)) +pr_err ("Failed to set tracing on: %s\n" , p); +if (v) +tracer_tracing_on (tr); +else +tracer_tracing_off (tr); + } + p =xbc_node_find_value (node,"trace_clock" ,NULL );if (p && *p !='\0' ) {if (tracing_set_clock (tr, p) <0 )
Masami Hiramatsu
maintainerPATCH v2 3/6 Documentation: tracing: Add tracing_on option to boot-time tracer
In reply to Masami HiramatsuAdd tracing_on option description to the boot-time tracer.
--- Documentation/trace/boottime-trace.rst | 4 ++++ 1 file changed, 4 insertions(+)
Documentation/trace/boottime-trace.rst +4 -0
@@ -61,6 +61,10 @@ These options can be used for each instance including global ftrace node. ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]] Enable given ftrace options. +ftrace.[instance.INSTANCE.]tracing_on = 0|1 + Enable/Disable tracing on this instance when boot. + (you can enable it by the "traceon" event trigger action) + ftrace.[instance.INSTANCE.]trace_clock = CLOCK Set given CLOCK to ftrace's trace_clock.
Masami Hiramatsu
maintainerPATCH v2 4/6 tracing/kprobes: Support perf-style return probe
In reply to Masami HiramatsuSupport perf-style return probe ("SYMBOL%return") in kprobe events. This will allow boot-time tracing user to define a return probe event.
--- kernel/trace/trace_kprobe.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
kernel/trace/trace_kprobe.c +20 -1
@@ -709,6 +709,18 @@ static inline void sanitize_event_name(char *name) *name ='_' ; } +static inline bool split_return_suffix (char *symbol) +{ +char *p =strchr (symbol,'%' ); + +if (p && !strcmp (p,"%return" )) { + *p ='\0' ; +return true; + } + +return false; +} +static int trace_kprobe_create (int argc,const char *argv[]) {/*
@@ -717,6 +729,9 @@ static int trace_kprobe_create(int argc, const char *argv[])* p[:[GRP/]EVENT] [MOD:]KSYM[+OFFS]|KADDR [FETCHARGS] * - Add kretprobe: * r[MAXACTIVE][:[GRP/]EVENT] [MOD:]KSYM[+0] [FETCHARGS] +* Or +* p:[GRP/]EVENT] [MOD:]KSYM[+0]%return [FETCHARGS] +* * Fetch args: * $retval : fetch return value * $stack : fetch stack address
@@ -746,7 +761,6 @@ static int trace_kprobe_create(int argc, const char *argv[])switch (argv[0 ][0 ]) {case 'r' : is_return = true; - flags |=TPARG_FL_RETURN ;break ;case 'p' :break ;
@@ -804,12 +818,17 @@ static int trace_kprobe_create(int argc, const char *argv[]) symbol =kstrdup (argv[1 ],GFP_KERNEL );if (!symbol)return -ENOMEM ; + + is_return =split_return_suffix (symbol) || is_return; +/* TODO: support .init module functions */ ret =traceprobe_split_symbol_offset (symbol, &offset);if (ret || offset <0 || offset >UINT_MAX ) {trace_probe_log_err (0 ,BAD_PROBE_ADDR ); goto parse_error; } +if (is_return) + flags |=TPARG_FL_RETURN ;if (kprobe_on_func_entry (NULL , symbol, offset)) flags |=TPARG_FL_FENTRY ;if (offset && is_return && !(flags &TPARG_FL_FENTRY )) {
Masami Hiramatsu
maintainerRe: [PATCH v2 4/6] tracing/kprobes: Support perf-style return probe
Quoted 3 lines · On Tue, 1 Sep 2020 16:57:10 +0900 Masami Hiramatsu <[email protected]> wrote:
Support perf-style return probe ("SYMBOL%return") in kprobe events. This will allow boot-time tracing user to define a return probe event.
Hmm, I think I should add this for uprobe event too. I'll update the series.
Thank you,
Quoted 65 lines
Signed-off-by Masami Hiramatsu <[email protected]>--- kernel/trace/trace_kprobe.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)kernel/trace/trace_kprobe.c +20
-1@@ -709,6 +709,18 @@ static inline void sanitize_event_name(char *name) *name ='_' ; } +static inline bool split_return_suffix (char *symbol) +{ +char *p =strchr (symbol,'%' ); + +if (p && !strcmp (p,"%return" )) { + *p ='\0' ; +return true; + } + +return false; +} +static int trace_kprobe_create (int argc,const char *argv[]) {/* @@ -717,6 +729,9 @@ static int trace_kprobe_create(int argc, const char *argv[])* p[:[GRP/]EVENT] [MOD:]KSYM[+OFFS]|KADDR [FETCHARGS] * - Add kretprobe: * r[MAXACTIVE][:[GRP/]EVENT] [MOD:]KSYM[+0] [FETCHARGS] +* Or +* p:[GRP/]EVENT] [MOD:]KSYM[+0]%return [FETCHARGS] +* * Fetch args: * $retval : fetch return value * $stack : fetch stack address @@ -746,7 +761,6 @@ static int trace_kprobe_create(int argc, const char *argv[])switch (argv[0 ][0 ]) {case 'r' : is_return = true; - flags |=TPARG_FL_RETURN ;break ;case 'p' :break ;@@ -804,12 +818,17 @@ static int trace_kprobe_create(int argc, const char *argv[]) symbol =kstrdup (argv[1 ],GFP_KERNEL );if (!symbol)return -ENOMEM ; + + is_return =split_return_suffix (symbol) || is_return; +/* TODO: support .init module functions */ ret =traceprobe_split_symbol_offset (symbol, &offset);if (ret || offset <0 || offset >UINT_MAX ) {trace_probe_log_err (0 ,BAD_PROBE_ADDR ); goto parse_error; } +if (is_return) + flags |=TPARG_FL_RETURN ;if (kprobe_on_func_entry (NULL , symbol, offset)) flags |=TPARG_FL_FENTRY ;if (offset && is_return && !(flags &TPARG_FL_FENTRY )) {
Signature · 1 line
Masami Hiramatsu <[email protected]>
Masami Hiramatsu
maintainerPATCH v2 5/6 Documentation: tracing: Add %return suffix description
In reply to Masami HiramatsuAdd a description of the %return suffix option for kprobe tracer.
---
Changes in v2
- Use a tab for indentation as other lines do.
--- Documentation/trace/kprobetrace.rst | 2 ++ 1 file changed, 2 insertions(+)
Documentation/trace/kprobetrace.rst +2 -0
@@ -30,6 +30,7 @@ Synopsis of kprobe_events p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe + p:[GRP/]EVENT] [MOD:]SYM[+0]%return [FETCHARGS] : Set a return probe -:[GRP/]EVENT : Clear a probe GRP : Group name. If omitted, use "kprobes" for it.
@@ -37,6 +38,7 @@ Synopsis of kprobe_events based on SYM+offs or MEMADDR. MOD : Module name which has given SYM. SYM[+offs] : Symbol+offset where the probe is inserted. + SYM%return : Return address of the symbol MEMADDR : Address where the probe is inserted. MAXACTIVE : Maximum number of instances of the specified function that can be probed simultaneously, or 0 for the default value
Masami Hiramatsu
maintainerPATCH v2 6/6 Documentation: tracing: boot: Add an example of tracing function-calls
In reply to Masami HiramatsuAdd an example of tracing function calls on a specific function.
--- Documentation/trace/boottime-trace.rst | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
Documentation/trace/boottime-trace.rst +20 -0
@@ -168,6 +168,26 @@ is for tracing functions starting with "user\_", and others tracing The instance node also accepts event nodes so that each instance can customize its event tracing. +With the trigger action and kprobes, you can trace function-graph while +a function is called. For example, this will trace all function calls in +the pci_proc_init():: + + ftrace { + tracing_on = 0 + tracer = function_graph + event.kprobes { + start_event { + probes = "pci_proc_init" + actions = "traceon" + } + end_event { + probes = "pci_proc_init%return" + actions = "traceoff" + } + } + } + + This boot-time tracing also supports ftrace kernel parameters via boot config. For example, following kernel parameters::