Skip to contentSkip to conversation

tracing/boot: Add new options for tracing specific period

PATCH Superseded Masami Hiramatsu 1 participant 8 messages Sep 1, 2020 – Sep 10, 2020

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

v2 cover tracing/boot: Add new options for tracing specific period

Hi,

Here is the 2nd version of the series to improve the boot-time tracing to support kretprobe and tracing_on option. Previous version is here:

https://lkml.kernel.org/r/159887792384.1330989.5993224243767476896.stgit@devnote2

This version fixes a build error ([1/6]) and fix a space indent issue ([5/6]).

The combination of tracing_on and kretprobe allows us to trace events while a specific function call period. For example, the below bootconfig will make a function callgraph in the pci_proc_init() function at boot time.

ftrace {
	tracing_on = 0  # off at start
	tracer = function_graph
	event.kprobes {
		start_event {
			probes = "pci_proc_init"
			actions = "traceon"
		}
		end_event {
			probes = "pci_proc_init%return"
			actions = "traceoff"
		}
	}
}

Here is the example output;

# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |

 0)               |  pci_proc_init() {
 0)               |    proc_mkdir() {
 0)               |      proc_mkdir_data() {
 0)               |        __proc_create() {
 0)               |          _raw_read_lock() {
 0)   0.179 us    |            preempt_count_add();
 0)   0.203 us    |            do_raw_read_lock();
 0)   1.210 us    |          }
 0)               |          __xlate_proc_name() {
 0)   0.449 us    |            pde_subdir_find();
 0)   0.913 us    |          }
 0)               |          _raw_read_unlock() {
 0)   0.169 us    |            do_raw_read_unlock();
 0)   0.175 us    |            preempt_count_sub();
 0)   0.841 us    |          }
 0)               |          kmem_cache_alloc() {
 0)               |            fs_reclaim_acquire() {
 0)   0.154 us    |              __need_fs_reclaim();
 0)   0.240 us    |              fs_reclaim_acquire.part.0();
 0)   0.889 us    |            }
 0)               |            fs_reclaim_release() {
 0)   0.174 us    |              __need_fs_reclaim();
 0)   0.516 us    |            }
 0)   0.157 us    |            should_failslab();
 0)               |            rcu_read_lock_sched_held() {
 0)               |              rcu_read_lock_held_common() {
 0)   0.156 us    |                rcu_is_watching();
 0)   0.158 us    |                rcu_lockdep_current_cpu_online();
 0)   0.735 us    |              }
 0)   1.054 us    |            }
 0)   3.407 us    |          }
 0)   0.168 us    |          __raw_spin_lock_init();
 0)   7.575 us    |        }
 0)               |        proc_register() {
 0)               |          _raw_spin_lock_irqsave() {
 0)   0.187 us    |            preempt_count_add();

...

Thank you,

---

Masami Hiramatsu (6):
kprobes: tracing/kprobes: Fix to kill kprobes on initmem after boot tracing/boot: Add per-instance tracing_on option support
Documentation: tracing: Add tracing_on option to boot-time tracer tracing/kprobes: Support perf-style return probe
Documentation: tracing: Add %return suffix description
Documentation: tracing: boot: Add an example of tracing function-calls

 Documentation/trace/boottime-trace.rst |   24 ++++++++++++++++++++++++
 Documentation/trace/kprobetrace.rst    |    2 ++
 include/linux/kprobes.h                |    5 +++++
 init/main.c                            |    2 ++
 kernel/kprobes.c                       |   22 ++++++++++++++++++++++
 kernel/trace/trace_boot.c              |   10 ++++++++++
 kernel/trace/trace_kprobe.c            |   21 ++++++++++++++++++++-
 7 files changed, 85 insertions(+), 1 deletion(-)

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

PATCH v2 1/6 kprobes: tracing/kprobes: Fix to kill kprobes on initmem after boot

Since kprobe_event= cmdline option allows user to put kprobes on the functions in initmem, kprobe has to make such probes gone after boot. Currently the probes on the init functions in modules will be handled by module callback, but the kernel init text isn't handled. Without this, kprobes may access non-exist text area to disable or remove it.

Fixes 970988e19eb0 ("tracing/kprobe: Add kprobe_event= boot parameter")

---
Changes in v2:
- Fix kprobe_free_init_mem() not depending on CONFIG_DEBUG_FS.

---
 include/linux/kprobes.h |    5 +++++
 init/main.c             |    2 ++
 kernel/kprobes.c        |   22 ++++++++++++++++++++++
 3 files changed, 29 insertions(+)
include/linux/kprobes.h +5 -0
index 9be1bff4f586..8aab327b5539 100644
@@ -373,6 +373,8 @@ void unregister_kretprobes(struct kretprobe **rps, int num);
 void kprobe_flush_task(struct task_struct *tk);
 void recycle_rp_inst(struct kretprobe_instance *ri, struct hlist_head *head);
 
+void kprobe_free_init_mem(void);
+
 int disable_kprobe(struct kprobe *kp);
 int enable_kprobe(struct kprobe *kp);
 
@@ -435,6 +437,9 @@ static inline void unregister_kretprobes(struct kretprobe **rps, int num)
 static inline void kprobe_flush_task(struct task_struct *tk)
 {
 }
+static inline void kprobe_free_init_mem(void)
+{
+}
 static inline int disable_kprobe(struct kprobe *kp)
 {
 	return -ENOSYS;
init/main.c +2 -0
index ae78fb68d231..038128b2a755 100644
@@ -33,6 +33,7 @@
 #include <linux/nmi.h>
 #include <linux/percpu.h>
 #include <linux/kmod.h>
+#include <linux/kprobes.h>
 #include <linux/vmalloc.h>
 #include <linux/kernel_stat.h>
 #include <linux/start_kernel.h>
@@ -1402,6 +1403,7 @@ static int __ref kernel_init(void *unused)
 	kernel_init_freeable();
 	/* need to finish all async __init code before freeing the memory */
 	async_synchronize_full();
+	kprobe_free_init_mem();
 	ftrace_free_init_mem();
 	free_initmem();
 	mark_readonly();
kernel/kprobes.c +22 -0
index 287b263c9cb9..2880cdf37c47 100644
@@ -2452,6 +2452,28 @@ static struct notifier_block kprobe_module_nb = {
 extern unsigned long __start_kprobe_blacklist[];
 extern unsigned long __stop_kprobe_blacklist[];
 
+void kprobe_free_init_mem(void)
+{
+	void *start = (void *)(&__init_begin);
+	void *end = (void *)(&__init_end);
+	struct hlist_head *head;
+	struct kprobe *p;
+	int i;
+
+	mutex_lock(&kprobe_mutex);
+
+	/* Kill all kprobes on initmem */
+	for (i = 0; i < KPROBE_TABLE_SIZE; i++) {
+		head = &kprobe_table[i];
+		hlist_for_each_entry(p, head, hlist) {
+			if (start <= (void *)p->addr && (void *)p->addr < end)
+				kill_kprobe(p);
+		}
+	}
+
+	mutex_unlock(&kprobe_mutex);
+}
+
 static int __init init_kprobes(void)
 {
 	int i, err = 0;
author maintainer
#2 lore raw

PATCH v2 2/6 tracing/boot: Add per-instance tracing_on option support

In reply to Masami Hiramatsu

Add per-instance tracing_on option, which will be useful with traceon/traceoff event trigger actions. For example, if we disable tracing_on by default and set traceon and traceoff on a pair of events, we can trace functions between the pair of events.

---
 kernel/trace/trace_boot.c |   10 ++++++++++
 1 file changed, 10 insertions(+)
kernel/trace/trace_boot.c +10 -0
index fa0fc08c6ef8..d52d441a17e8 100644
@@ -40,6 +40,16 @@ trace_boot_set_instance_options(struct trace_array *tr, struct xbc_node *node)
 			pr_err("Failed to set option: %s\n", buf);
 	}
 
+	p = xbc_node_find_value(node, "tracing_on", NULL);
+	if (p && *p != '\0') {
+		if (kstrtoul(p, 10, &v))
+			pr_err("Failed to set tracing on: %s\n", p);
+		if (v)
+			tracer_tracing_on(tr);
+		else
+			tracer_tracing_off(tr);
+	}
+
 	p = xbc_node_find_value(node, "trace_clock", NULL);
 	if (p && *p != '\0') {
 		if (tracing_set_clock(tr, p) < 0)
author maintainer
#3 lore raw

PATCH v2 3/6 Documentation: tracing: Add tracing_on option to boot-time tracer

In reply to Masami Hiramatsu

Add tracing_on option description to the boot-time tracer.

---
 Documentation/trace/boottime-trace.rst |    4 ++++
 1 file changed, 4 insertions(+)
Documentation/trace/boottime-trace.rst +4 -0
index dcb390075ca1..1341b449acaa 100644
@@ -61,6 +61,10 @@ These options can be used for each instance including global ftrace node.
 ftrace.[instance.INSTANCE.]options = OPT1[, OPT2[...]]
    Enable given ftrace options.
 
+ftrace.[instance.INSTANCE.]tracing_on = 0|1
+   Enable/Disable tracing on this instance when boot.
+   (you can enable it by the "traceon" event trigger action)
+
 ftrace.[instance.INSTANCE.]trace_clock = CLOCK
    Set given CLOCK to ftrace's trace_clock.
 
author maintainer
#4 lore raw

PATCH v2 4/6 tracing/kprobes: Support perf-style return probe

In reply to Masami Hiramatsu

Support perf-style return probe ("SYMBOL%return") in kprobe events. This will allow boot-time tracing user to define a return probe event.

---
 kernel/trace/trace_kprobe.c |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
kernel/trace/trace_kprobe.c +20 -1
index aefb6065b508..391361b67c8f 100644
@@ -709,6 +709,18 @@ static inline void sanitize_event_name(char *name)
 			*name = '_';
 }
 
+static inline bool split_return_suffix(char *symbol)
+{
+	char *p = strchr(symbol, '%');
+
+	if (p && !strcmp(p, "%return")) {
+		*p = '\0';
+		return true;
+	}
+
+	return false;
+}
+
 static int trace_kprobe_create(int argc, const char *argv[])
 {
 	/*
@@ -717,6 +729,9 @@ static int trace_kprobe_create(int argc, const char *argv[])
 	 *      p[:[GRP/]EVENT] [MOD:]KSYM[+OFFS]|KADDR [FETCHARGS]
 	 *  - Add kretprobe:
 	 *      r[MAXACTIVE][:[GRP/]EVENT] [MOD:]KSYM[+0] [FETCHARGS]
+	 *    Or
+	 *      p:[GRP/]EVENT] [MOD:]KSYM[+0]%return [FETCHARGS]
+	 *
 	 * Fetch args:
 	 *  $retval	: fetch return value
 	 *  $stack	: fetch stack address
@@ -746,7 +761,6 @@ static int trace_kprobe_create(int argc, const char *argv[])
 	switch (argv[0][0]) {
 	case 'r':
 		is_return = true;
-		flags |= TPARG_FL_RETURN;
 		break;
 	case 'p':
 		break;
@@ -804,12 +818,17 @@ static int trace_kprobe_create(int argc, const char *argv[])
 		symbol = kstrdup(argv[1], GFP_KERNEL);
 		if (!symbol)
 			return -ENOMEM;
+
+		is_return = split_return_suffix(symbol) || is_return;
+
 		/* TODO: support .init module functions */
 		ret = traceprobe_split_symbol_offset(symbol, &offset);
 		if (ret || offset < 0 || offset > UINT_MAX) {
 			trace_probe_log_err(0, BAD_PROBE_ADDR);
 			goto parse_error;
 		}
+		if (is_return)
+			flags |= TPARG_FL_RETURN;
 		if (kprobe_on_func_entry(NULL, symbol, offset))
 			flags |= TPARG_FL_FENTRY;
 		if (offset && is_return && !(flags & TPARG_FL_FENTRY)) {
author maintainer
#5 lore raw

Re: [PATCH v2 4/6] tracing/kprobes: Support perf-style return probe

Quoted 3 lines · On Tue, 1 Sep 2020 16:57:10 +0900 Masami Hiramatsu <[email protected]> wrote:

Support perf-style return probe ("SYMBOL%return") in kprobe events. This will allow boot-time tracing user to define a return probe event.

Hmm, I think I should add this for uprobe event too. I'll update the series.

Thank you,

Quoted 65 lines
---
 kernel/trace/trace_kprobe.c |   21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)
kernel/trace/trace_kprobe.c +20 -1
index aefb6065b508..391361b67c8f 100644
@@ -709,6 +709,18 @@ static inline void sanitize_event_name(char *name)
 			*name = '_';
 }
 
+static inline bool split_return_suffix(char *symbol)
+{
+	char *p = strchr(symbol, '%');
+
+	if (p && !strcmp(p, "%return")) {
+		*p = '\0';
+		return true;
+	}
+
+	return false;
+}
+
 static int trace_kprobe_create(int argc, const char *argv[])
 {
 	/*
@@ -717,6 +729,9 @@ static int trace_kprobe_create(int argc, const char *argv[])
 	 *      p[:[GRP/]EVENT] [MOD:]KSYM[+OFFS]|KADDR [FETCHARGS]
 	 *  - Add kretprobe:
 	 *      r[MAXACTIVE][:[GRP/]EVENT] [MOD:]KSYM[+0] [FETCHARGS]
+	 *    Or
+	 *      p:[GRP/]EVENT] [MOD:]KSYM[+0]%return [FETCHARGS]
+	 *
 	 * Fetch args:
 	 *  $retval	: fetch return value
 	 *  $stack	: fetch stack address
@@ -746,7 +761,6 @@ static int trace_kprobe_create(int argc, const char *argv[])
 	switch (argv[0][0]) {
 	case 'r':
 		is_return = true;
-		flags |= TPARG_FL_RETURN;
 		break;
 	case 'p':
 		break;
@@ -804,12 +818,17 @@ static int trace_kprobe_create(int argc, const char *argv[])
 		symbol = kstrdup(argv[1], GFP_KERNEL);
 		if (!symbol)
 			return -ENOMEM;
+
+		is_return = split_return_suffix(symbol) || is_return;
+
 		/* TODO: support .init module functions */
 		ret = traceprobe_split_symbol_offset(symbol, &offset);
 		if (ret || offset < 0 || offset > UINT_MAX) {
 			trace_probe_log_err(0, BAD_PROBE_ADDR);
 			goto parse_error;
 		}
+		if (is_return)
+			flags |= TPARG_FL_RETURN;
 		if (kprobe_on_func_entry(NULL, symbol, offset))
 			flags |= TPARG_FL_FENTRY;
 		if (offset && is_return && !(flags & TPARG_FL_FENTRY)) {
Signature · 1 line
Masami Hiramatsu <[email protected]>
author maintainer
#6 lore raw

PATCH v2 5/6 Documentation: tracing: Add %return suffix description

In reply to Masami Hiramatsu

Add a description of the %return suffix option for kprobe tracer.

---
Changes in v2
- Use a tab for indentation as other lines do.

---
 Documentation/trace/kprobetrace.rst |    2 ++
 1 file changed, 2 insertions(+)
Documentation/trace/kprobetrace.rst +2 -0
index c1709165c553..a18eb97a263c 100644
@@ -30,6 +30,7 @@ Synopsis of kprobe_events
 
   p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS]	: Set a probe
   r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS]	: Set a return probe
+  p:[GRP/]EVENT] [MOD:]SYM[+0]%return [FETCHARGS]	: Set a return probe
   -:[GRP/]EVENT						: Clear a probe
 
  GRP		: Group name. If omitted, use "kprobes" for it.
@@ -37,6 +38,7 @@ Synopsis of kprobe_events
 		  based on SYM+offs or MEMADDR.
  MOD		: Module name which has given SYM.
  SYM[+offs]	: Symbol+offset where the probe is inserted.
+ SYM%return	: Return address of the symbol
  MEMADDR	: Address where the probe is inserted.
  MAXACTIVE	: Maximum number of instances of the specified function that
 		  can be probed simultaneously, or 0 for the default value
author maintainer
#7 lore raw

PATCH v2 6/6 Documentation: tracing: boot: Add an example of tracing function-calls

In reply to Masami Hiramatsu

Add an example of tracing function calls on a specific function.

---
 Documentation/trace/boottime-trace.rst |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
Documentation/trace/boottime-trace.rst +20 -0
index 1341b449acaa..c216f5695ae2 100644
@@ -168,6 +168,26 @@ is for tracing functions starting with "user\_", and others tracing
 The instance node also accepts event nodes so that each instance
 can customize its event tracing.
 
+With the trigger action and kprobes, you can trace function-graph while
+a function is called. For example, this will trace all function calls in
+the pci_proc_init()::
+
+  ftrace {
+        tracing_on = 0
+        tracer = function_graph
+        event.kprobes {
+                start_event {
+                        probes = "pci_proc_init"
+                        actions = "traceon"
+                }
+                end_event {
+                        probes = "pci_proc_init%return"
+                        actions = "traceoff"
+                }
+        }
+  }
+
+
 This boot-time tracing also supports ftrace kernel parameters via boot
 config.
 For example, following kernel parameters::