Skip to contentSkip to conversation

perf: ftrace enhancement

PATCH Applied Changbin Du 5 participants 58 messages May 10, 2020 – Jun 25, 2020 perf events

v1 of 8 Applied Latest is v8 →

lore
Share & tools
lore
#0 lore raw

cover perf: ftrace enhancement

The perf has basic kernel ftrace support but lack support of most tracing options. This serias is target to enhance the perf ftrace functionality so that we can make full use of kernel ftrace with only perf.

In general, this serias be cataloged into two main changes:
1) Improve usability of existing functions. For example, we don't need to type extra option to select the tracer.
2) Add new options to support all other ftrace functions.

Here is a glance of all ftrace functions with this serias:
* - improved existing options.
+ - new added options.

$ sudo perf ftrace -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- <command> [<options>]

* -a, --all-cpus system-wide collection from all CPUs
+ -b, --buffer-size <n>
size of per cpu buffer in kb
-C, --cpu <cpu> list of cpus to monitor
+ -d, --delay <n> Wait <n> ms before tracing
-D, --graph-depth <n>
Max depth for function graph tracer
* -G, --graph-funcs <func>
Set graph filter on given functions (imply to use function_graph tracer)
-g, --nograph-funcs <func>
Set nograph filter on given functions (imply to use function_graph tracer)

+   -L, --list-functions  List available functions to filter
+   -l, --long-info       Show process names, PIDs, timestamps, irq-info if available

-N, --notrace-funcs <func>
do not trace given functions
+ -P, --no-pager Do not use pager
-p, --pid <pid> trace on existing process id
+ -s, --func-stack-trace
Show kernel stack trace for function tracer
+ -t, --tid <tid> trace on existing thread id (exclusive to --pid)
-T, --trace-funcs <func>
trace given functions only

+   -u, --userstacktrace  Show stacktrace of the current user space thread
    -v, --verbose         be more verbose
+       --funcgraph-tail  Show function tails comment (function_graph only)
+       --latency-format  displays additional information about the latency (function_graph only)
+       --nofuncgraph-irqs
                          Ignore functions that happen inside interrupt (function_graph only)
+       --nosleep-time    Measure on-CPU time only (function_graph only)
+       --trace-children  Trace children processes
+       --tracing-thresh <n>

Only show functions of which the duration is greater than <n>µs

Changbin Du (19):
perf ftrace: trace system wide if no target is given
perf ftrace: detect workload failure
perf ftrace: select function/function_graph tracer automatically perf ftrace: add support for tracing option 'func_stack_trace' perf ftrace: add option '-l/--list-functions' to list available
functions
perf ftrace: add support for trace option sleep-time
perf ftrace: add support for trace option funcgraph-irqs perf ftrace: add option -l/--long-info to show more info perf ftrace: add support for trace option tracing_thresh perf ftrace: add support for trace option funcgraph-tail perf ftrace: add option '-u/--userstacktrace' to show userspace
stacktrace
perf ftrace: add support for tracing children processes perf ftrace: add option '-b/--buffer-size' to set per-cpu buffer size perf ftrace: add option -P/--no-pager to disable pager perf ftrace: show trace column header
perf ftrace: add option -t/--tid to filter by thread id perf ftrace: add option -d/--delay to delay tracing
perf ftrace: add option --latency-format to display more info about
delay
perf ftrace: add change log

 tools/perf/Documentation/perf-config.txt |   5 -
 tools/perf/builtin-ftrace.c              | 388 ++++++++++++++++++++---
 2 files changed, 345 insertions(+), 48 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH 1/19 perf ftrace: trace system wide if no target is given

This align ftrace to other perf sub-commands that if no target specified then we trace all functions.

---
 tools/perf/builtin-ftrace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-ftrace.c +1 -1
index d5adc417a4ca..11fc02037899 100644
@@ -493,7 +493,7 @@ int cmd_ftrace(int argc, const char **argv)
 	argc = parse_options(argc, argv, ftrace_options, ftrace_usage,
 			    PARSE_OPT_STOP_AT_NON_OPTION);
 	if (!argc && target__none(&ftrace.target))
-		usage_with_options(ftrace_usage, ftrace_options);
+		ftrace.target.system_wide = true;
 
 	ret = target__validate(&ftrace.target);
 	if (ret) {
Signature · 1 line
2.25.1
maintainer Applied
#2 lore raw

Re: [PATCH 01/19] perf ftrace: trace system wide if no target is given

Em Sun, May 10, 2020 at 11:06:10PM +0800, Changbin Du escreveu:

Quoted 2 lines

This align ftrace to other perf sub-commands that if no target specified then we trace all functions.

Thanks, applied,

- Arnaldo

Quoted 21 lines
---
 tools/perf/builtin-ftrace.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/perf/builtin-ftrace.c +1 -1
index d5adc417a4ca..11fc02037899 100644
@@ -493,7 +493,7 @@ int cmd_ftrace(int argc, const char **argv)
 	argc = parse_options(argc, argv, ftrace_options, ftrace_usage,
 			    PARSE_OPT_STOP_AT_NON_OPTION);
 	if (!argc && target__none(&ftrace.target))
-		usage_with_options(ftrace_usage, ftrace_options);
+		ftrace.target.system_wide = true;
 
 	ret = target__validate(&ftrace.target);
 	if (ret) {
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#3 lore raw

PATCH 2/19 perf ftrace: detect workload failure

In reply to Changbin Du

Currently there's no error message prompted if we failed to start workload. And we still get some trace which is confusing. Let's tell users what happened.

---
 tools/perf/builtin-ftrace.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
tools/perf/builtin-ftrace.c +11 -2
index 11fc02037899..5584f8dec25d 100644
@@ -45,6 +45,7 @@ struct filter_entry {
 	char			name[];
 };
 
+static volatile int workload_exec_errno;
 static bool done;
 
 static void sig_handler(int sig __maybe_unused)
@@ -63,7 +64,7 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused,
 						siginfo_t *info __maybe_unused,
 						void *ucontext __maybe_unused)
 {
-	/* workload_exec_errno = info->si_value.sival_int; */
+	workload_exec_errno = info->si_value.sival_int;
 	done = true;
 }
 
@@ -382,6 +383,14 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 
 	write_tracing_file("tracing_on", "0");
 
+	if (workload_exec_errno) {
+		const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+		/* flush stdout first so below error msg appears at the end. */
+		fflush(stdout);
+		pr_err("workload failed: %s\n", emsg);
+		goto out_close_fd;
+	}
+
 	/* read remaining buffer contents */
 	while (true) {
 		int n = read(trace_fd, buf, sizeof(buf));
@@ -396,7 +405,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 out_reset:
 	reset_tracing_files(ftrace);
 out:
-	return done ? 0 : -1;
+	return (done && !workload_exec_errno) ? 0 : -1;
 }
 
 static int perf_ftrace_config(const char *var, const char *value, void *cb)
Signature · 1 line
2.25.1
maintainer Tested-by on v1 2/19 Applied
#4 lore raw

Re: [PATCH 02/19] perf ftrace: detect workload failure

Em Sun, May 10, 2020 at 11:06:11PM +0800, Changbin Du escreveu:

Quoted 3 lines

Currently there's no error message prompted if we failed to start workload. And we still get some trace which is confusing. Let's tell users what happened.

Applied, but please in the future add Before and After output of the commands for these to be clear even to my 4yo kid:

commit 2a5193e460eaf8792a87f678e3c5f78c88db2123

Author: Changbin Du <[email protected]>
Date:   Sun May 10 23:06:11 2020 +0800

perf ftrace: Detect workload failure

Currently there's no error message prompted if we failed to start workload. And we still get some trace which is confusing. Let's tell users what happened.

Committer testing:

Before:

# perf ftrace nonsense |& head
5) | switch_mm_irqs_off() {
5) 0.400 us | load_new_mm_cr3();
5) 3.261 us | }
------------------------------------------
5) <idle>-0 => <...>-3494
------------------------------------------

5) | finish_task_switch() {
5) ==========> |
5) | smp_irq_work_interrupt() {
# type nonsense
-bash: type: nonsense: not found
#

After:

# perf ftrace nonsense |& head
workload failed: No such file or directory
# type nonsense
-bash: type: nonsense: not found
#

tools/perf/builtin-ftrace.c +11 -2
index b7d3fb5fa8b1..2bfc1b0db536 100644
@@ -45,6 +45,7 @@ struct filter_entry {
 	char			name[];
 };
 
+static volatile int workload_exec_errno;
 static bool done;
 
 static void sig_handler(int sig __maybe_unused)
@@ -63,7 +64,7 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused,
 						siginfo_t *info __maybe_unused,
 						void *ucontext __maybe_unused)
 {
-	/* workload_exec_errno = info->si_value.sival_int; */
+	workload_exec_errno = info->si_value.sival_int;
 	done = true;
 }
 
@@ -383,6 +384,14 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 
 	write_tracing_file("tracing_on", "0");
 
+	if (workload_exec_errno) {
+		const char *emsg = str_error_r(workload_exec_errno, buf, sizeof(buf));
+		/* flush stdout first so below error msg appears at the end. */
+		fflush(stdout);
+		pr_err("workload failed: %s\n", emsg);
+		goto out_close_fd;
+	}
+
 	/* read remaining buffer contents */
 	while (true) {
 		int n = read(trace_fd, buf, sizeof(buf));
@@ -397,7 +406,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 out_reset:
 	reset_tracing_files(ftrace);
 out:
-	return done ? 0 : -1;
+	return (done && !workload_exec_errno) ? 0 : -1;
 }
 
 static int perf_ftrace_config(const char *var, const char *value, void *cb)
#5 lore raw

PATCH 3/19 perf ftrace: select function/function_graph tracer automatically

In reply to Changbin Du

The '-g/-G' options have already implied function_graph tracer should be used instead of function tracer. So the extra option '--tracer' can be killed.

This patch changes the behavior as below:
- By default, function tracer is used.
- If '-g' or '-G' option is on, then function_graph tracer is used.
- The perf configuration item 'ftrace.tracer' is removed.
- The default filter for -G is to trace all functions.

Here are some examples.

This will start tracing all functions using function tracer:

  $ sudo perf ftrace

This will trace all functions using function graph tracer:

  $ sudo perf ftrace -G

This will trace function vfs_read using function graph tracer:

  $ sudo perf ftrace -G vfs_read
---
 tools/perf/Documentation/perf-config.txt |  5 ---
 tools/perf/builtin-ftrace.c              | 39 ++++++------------------
 2 files changed, 9 insertions(+), 35 deletions(-)
tools/perf/Documentation/perf-config.txt +0 -5
index f16d8a71d3f5..fad6c48ed76a 100644
@@ -612,11 +612,6 @@ trace.*::
 		"libbeauty", the default, to use the same argument beautifiers used in the
 		strace-like sys_enter+sys_exit lines.
 
-ftrace.*::
-	ftrace.tracer::
-		Can be used to select the default tracer. Possible values are
-		'function' and 'function_graph'.
-
 llvm.*::
 	llvm.clang-path::
 		Path to clang. If omit, search it from $PATH.
tools/perf/builtin-ftrace.c +9 -30
index 5584f8dec25d..57e656c35d28 100644
@@ -27,7 +27,7 @@
 #include "util/cap.h"
 #include "util/config.h"
 
-#define DEFAULT_TRACER  "function_graph"
+#define DEFAULT_TRACER  "function"
 
 struct perf_ftrace {
 	struct evlist		*evlist;
@@ -408,26 +408,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	return (done && !workload_exec_errno) ? 0 : -1;
 }
 
-static int perf_ftrace_config(const char *var, const char *value, void *cb)
-{
-	struct perf_ftrace *ftrace = cb;
-
-	if (!strstarts(var, "ftrace."))
-		return 0;
-
-	if (strcmp(var, "ftrace.tracer"))
-		return -1;
-
-	if (!strcmp(value, "function_graph") ||
-	    !strcmp(value, "function")) {
-		ftrace->tracer = value;
-		return 0;
-	}
-
-	pr_err("Please select \"function_graph\" (default) or \"function\"\n");
-	return -1;
-}
-
 static int parse_filter_func(const struct option *opt, const char *str,
 			     int unset __maybe_unused)
 {
@@ -467,8 +447,6 @@ int cmd_ftrace(int argc, const char **argv)
 		NULL
 	};
 	const struct option ftrace_options[] = {
-	OPT_STRING('t', "tracer", &ftrace.tracer, "tracer",
-		   "tracer to use: function_graph(default) or function"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
 	OPT_INCR('v', "verbose", &verbose,
@@ -481,10 +459,12 @@ int cmd_ftrace(int argc, const char **argv)
 		     "trace given functions only", parse_filter_func),
 	OPT_CALLBACK('N', "notrace-funcs", &ftrace.notrace, "func",
 		     "do not trace given functions", parse_filter_func),
-	OPT_CALLBACK('G', "graph-funcs", &ftrace.graph_funcs, "func",
-		     "Set graph filter on given functions", parse_filter_func),
+	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
+		     "Set graph filter on given functions (imply to use function_graph tracer)",
+		     parse_filter_func, "*"),
 	OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
-		     "Set nograph filter on given functions", parse_filter_func),
+		     "Set nograph filter on given functions (imply to use function_graph tracer)",
+		     parse_filter_func),
 	OPT_INTEGER('D', "graph-depth", &ftrace.graph_depth,
 		    "Max depth for function graph tracer"),
 	OPT_END()
@@ -495,15 +475,14 @@ int cmd_ftrace(int argc, const char **argv)
 	INIT_LIST_HEAD(&ftrace.graph_funcs);
 	INIT_LIST_HEAD(&ftrace.nograph_funcs);
 
-	ret = perf_config(perf_ftrace_config, &ftrace);
-	if (ret < 0)
-		return -1;
-
 	argc = parse_options(argc, argv, ftrace_options, ftrace_usage,
 			    PARSE_OPT_STOP_AT_NON_OPTION);
 	if (!argc && target__none(&ftrace.target))
 		ftrace.target.system_wide = true;
 
+	if (!list_empty(&ftrace.graph_funcs) || !list_empty(&ftrace.nograph_funcs))
+		ftrace.tracer = "function_graph";
+
 	ret = target__validate(&ftrace.target);
 	if (ret) {
 		char errbuf[512];
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 03/19] perf ftrace: select function/function_graph tracer automatically

Em Sun, May 10, 2020 at 11:06:12PM +0800, Changbin Du escreveu:

Quoted 3 lines

The '-g/-G' options have already implied function_graph tracer should be used instead of function tracer. So the extra option '--tracer' can be killed.

Well, since it is there, lets not remove it, you can even add a "Deprecated" message to it and then after some time, remove it.

Also, in this area, -g is used for asking for callchains in other perf tools, so its a candidate for a change in behaviour, as well with some deprecation warning for a while, using --call-graph for a while, etc.

I.e. we want it to have a similar set of options as the other perf tools, so that switching from, say:

  # perf trace <options> workload

To:

  # perf ftrace <options> workload

Would provide a similar experience, only that the first is limited to syscalls, the other to kernel functions.

- Arnaldo

Quoted 126 lines

This patch changes the behavior as below:
- By default, function tracer is used.
- If '-g' or '-G' option is on, then function_graph tracer is used.
- The perf configuration item 'ftrace.tracer' is removed.
- The default filter for -G is to trace all functions.

Here are some examples.

This will start tracing all functions using function tracer:

  $ sudo perf ftrace

This will trace all functions using function graph tracer:

  $ sudo perf ftrace -G

This will trace function vfs_read using function graph tracer:

  $ sudo perf ftrace -G vfs_read
---
 tools/perf/Documentation/perf-config.txt |  5 ---
 tools/perf/builtin-ftrace.c              | 39 ++++++------------------
 2 files changed, 9 insertions(+), 35 deletions(-)
tools/perf/Documentation/perf-config.txt +0 -5
index f16d8a71d3f5..fad6c48ed76a 100644
@@ -612,11 +612,6 @@ trace.*::
 		"libbeauty", the default, to use the same argument beautifiers used in the
 		strace-like sys_enter+sys_exit lines.
 
-ftrace.*::
-	ftrace.tracer::
-		Can be used to select the default tracer. Possible values are
-		'function' and 'function_graph'.
-
 llvm.*::
 	llvm.clang-path::
 		Path to clang. If omit, search it from $PATH.
tools/perf/builtin-ftrace.c +9 -30
index 5584f8dec25d..57e656c35d28 100644
@@ -27,7 +27,7 @@
 #include "util/cap.h"
 #include "util/config.h"
 
-#define DEFAULT_TRACER  "function_graph"
+#define DEFAULT_TRACER  "function"
 
 struct perf_ftrace {
 	struct evlist		*evlist;
@@ -408,26 +408,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	return (done && !workload_exec_errno) ? 0 : -1;
 }
 
-static int perf_ftrace_config(const char *var, const char *value, void *cb)
-{
-	struct perf_ftrace *ftrace = cb;
-
-	if (!strstarts(var, "ftrace."))
-		return 0;
-
-	if (strcmp(var, "ftrace.tracer"))
-		return -1;
-
-	if (!strcmp(value, "function_graph") ||
-	    !strcmp(value, "function")) {
-		ftrace->tracer = value;
-		return 0;
-	}
-
-	pr_err("Please select \"function_graph\" (default) or \"function\"\n");
-	return -1;
-}
-
 static int parse_filter_func(const struct option *opt, const char *str,
 			     int unset __maybe_unused)
 {
@@ -467,8 +447,6 @@ int cmd_ftrace(int argc, const char **argv)
 		NULL
 	};
 	const struct option ftrace_options[] = {
-	OPT_STRING('t', "tracer", &ftrace.tracer, "tracer",
-		   "tracer to use: function_graph(default) or function"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
 	OPT_INCR('v', "verbose", &verbose,
@@ -481,10 +459,12 @@ int cmd_ftrace(int argc, const char **argv)
 		     "trace given functions only", parse_filter_func),
 	OPT_CALLBACK('N', "notrace-funcs", &ftrace.notrace, "func",
 		     "do not trace given functions", parse_filter_func),
-	OPT_CALLBACK('G', "graph-funcs", &ftrace.graph_funcs, "func",
-		     "Set graph filter on given functions", parse_filter_func),
+	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
+		     "Set graph filter on given functions (imply to use function_graph tracer)",
+		     parse_filter_func, "*"),
 	OPT_CALLBACK('g', "nograph-funcs", &ftrace.nograph_funcs, "func",
-		     "Set nograph filter on given functions", parse_filter_func),
+		     "Set nograph filter on given functions (imply to use function_graph tracer)",
+		     parse_filter_func),
 	OPT_INTEGER('D', "graph-depth", &ftrace.graph_depth,
 		    "Max depth for function graph tracer"),
 	OPT_END()
@@ -495,15 +475,14 @@ int cmd_ftrace(int argc, const char **argv)
 	INIT_LIST_HEAD(&ftrace.graph_funcs);
 	INIT_LIST_HEAD(&ftrace.nograph_funcs);
 
-	ret = perf_config(perf_ftrace_config, &ftrace);
-	if (ret < 0)
-		return -1;
-
 	argc = parse_options(argc, argv, ftrace_options, ftrace_usage,
 			    PARSE_OPT_STOP_AT_NON_OPTION);
 	if (!argc && target__none(&ftrace.target))
 		ftrace.target.system_wide = true;
 
+	if (!list_empty(&ftrace.graph_funcs) || !list_empty(&ftrace.nograph_funcs))
+		ftrace.tracer = "function_graph";
+
 	ret = target__validate(&ftrace.target);
 	if (ret) {
 		char errbuf[512];
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
maintainer
#7 lore raw

Re: [PATCH 03/19] perf ftrace: select function/function_graph tracer automatically

In reply to Changbin Du

Hello,

Quoted 7 lines · Changbin Du

The '-g/-G' options have already implied function_graph tracer should be used instead of function tracer. So the extra option '--tracer' can be killed.

This patch changes the behavior as below:
- By default, function tracer is used.

Why? You also removed -t/--tracer option.

Quoted 1 line

- If '-g' or '-G' option is on, then function_graph tracer is used.

I'm ok with this.

Quoted 1 line

- The perf configuration item 'ftrace.tracer' is removed.

Why?

Quoted 1 line

- The default filter for -G is to trace all functions.

There's no reason to use -G option then. Also It might be confusing if it sees an argument - whether it's a function or a workload.

I just don't know why you want to change this as we have a way to change the default tracer already.

Thanks

Namhyung

#8 lore raw

Re: [PATCH 03/19] perf ftrace: select function/function_graph tracer automatically

Quoted 4 lines · Namhyung Kim

Hello,

Quoted 1 line · Changbin Du
Quoted 9 lines
Quoted 6 lines

The '-g/-G' options have already implied function_graph tracer should be used instead of function tracer. So the extra option '--tracer' can be killed.

This patch changes the behavior as below:
- By default, function tracer is used.

Why? You also removed -t/--tracer option.

To save typing. By this way, we do not need the -t/--tracer option which is redundant. When -G is given, it means graph tracer should be used. If not, function tracer.

Quoted 8 lines
Quoted 1 line

- If '-g' or '-G' option is on, then function_graph tracer is used.

I'm ok with this.

Quoted 1 line

- The perf configuration item 'ftrace.tracer' is removed.

Why?

As explained above, we don't need this option any longer.

Quoted 11 lines
Quoted 1 line

- The default filter for -G is to trace all functions.

There's no reason to use -G option then. Also It might be confusing if it sees an argument - whether it's a function or a workload.

I just don't know why you want to change this as we have a way to change the default tracer already.

Thanks

Namhyung

Signature · 2 lines
Cheers,
Changbin Du
#9 lore raw

PATCH 4/19 perf ftrace: add support for tracing option 'func_stack_trace'

In reply to Changbin Du

This adds support to display call trace for function tracer. To do this, just specify a '-s' option.

$ sudo perf ftrace -T vfs_read -s
 iio-sensor-prox-855   [003]   6168.369657: vfs_read <-ksys_read
 iio-sensor-prox-855   [003]   6168.369677: <stack trace>
 => vfs_read
 => ksys_read
 => __x64_sys_read
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 ...
---
 tools/perf/builtin-ftrace.c | 38 +++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
tools/perf/builtin-ftrace.c +38 -0
index 57e656c35d28..1d30c2d5f88b 100644
@@ -38,6 +38,7 @@ struct perf_ftrace {
 	struct list_head	graph_funcs;
 	struct list_head	nograph_funcs;
 	int			graph_depth;
+	bool			func_stack_trace;
 };
 
 struct filter_entry {
@@ -128,9 +129,27 @@ static int append_tracing_file(const char *name, const char *val)
 	return __write_tracing_file(name, val, true);
 }
 
+static int write_tracing_option_file(const char *name, const char *val)
+{
+	char *file;
+	int ret;
+
+	if (asprintf(&file, "options/%s", name) < 0)
+		return -1;
+
+	ret = __write_tracing_file(file, val, false);
+	free(file);
+	return ret;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
+static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
+{
+	write_tracing_option_file("func_stack_trace", "0");
+}
+
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 {
 	if (write_tracing_file("tracing_on", "0") < 0)
@@ -149,6 +168,7 @@ static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 		return -1;
 
 	reset_tracing_filters();
+	reset_tracing_options(ftrace);
 	return 0;
 }
 
@@ -204,6 +224,17 @@ static int set_tracing_cpu(struct perf_ftrace *ftrace)
 	return set_tracing_cpumask(cpumap);
 }
 
+static int set_tracing_func_stack_trace(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->func_stack_trace)
+		return 0;
+
+	if (write_tracing_option_file("func_stack_trace", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void)
 {
 	struct perf_cpu_map *cpumap = perf_cpu_map__new(NULL);
@@ -326,6 +357,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_func_stack_trace(ftrace) < 0) {
+		pr_err("failed to set tracing option func_stack_trace\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_filters(ftrace) < 0) {
 		pr_err("failed to set tracing filters\n");
 		goto out_reset;
@@ -459,6 +495,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "trace given functions only", parse_filter_func),
 	OPT_CALLBACK('N', "notrace-funcs", &ftrace.notrace, "func",
 		     "do not trace given functions", parse_filter_func),
+	OPT_BOOLEAN('s', "func-stack-trace", &ftrace.func_stack_trace,
+		    "Show kernel stack trace for function tracer"),
 	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
 		     "Set graph filter on given functions (imply to use function_graph tracer)",
 		     parse_filter_func, "*"),
Signature · 1 line
2.25.1
#10 lore raw

Re: [PATCH 04/19] perf ftrace: add support for tracing option 'func_stack_trace'

Em Sun, May 10, 2020 at 11:06:13PM +0800, Changbin Du escreveu:

Quoted 12 lines

This adds support to display call trace for function tracer. To do this, just specify a '-s' option.

$ sudo perf ftrace -T vfs_read -s
 iio-sensor-prox-855   [003]   6168.369657: vfs_read <-ksys_read
 iio-sensor-prox-855   [003]   6168.369677: <stack trace>
 => vfs_read
 => ksys_read
 => __x64_sys_read
 => do_syscall_64
 => entry_SYSCALL_64_after_hwframe
 ...

So, for the reason stated in my response to the patch 03/19, I think we should start here with --call-graph for this, instead of --func-stack-trace. And for now don't provide a one-letter equivalent.

- Arnaldo

Quoted 95 lines
---
 tools/perf/builtin-ftrace.c | 38 +++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
tools/perf/builtin-ftrace.c +38 -0
index 57e656c35d28..1d30c2d5f88b 100644
@@ -38,6 +38,7 @@ struct perf_ftrace {
 	struct list_head	graph_funcs;
 	struct list_head	nograph_funcs;
 	int			graph_depth;
+	bool			func_stack_trace;
 };
 
 struct filter_entry {
@@ -128,9 +129,27 @@ static int append_tracing_file(const char *name, const char *val)
 	return __write_tracing_file(name, val, true);
 }
 
+static int write_tracing_option_file(const char *name, const char *val)
+{
+	char *file;
+	int ret;
+
+	if (asprintf(&file, "options/%s", name) < 0)
+		return -1;
+
+	ret = __write_tracing_file(file, val, false);
+	free(file);
+	return ret;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
+static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
+{
+	write_tracing_option_file("func_stack_trace", "0");
+}
+
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 {
 	if (write_tracing_file("tracing_on", "0") < 0)
@@ -149,6 +168,7 @@ static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 		return -1;
 
 	reset_tracing_filters();
+	reset_tracing_options(ftrace);
 	return 0;
 }
 
@@ -204,6 +224,17 @@ static int set_tracing_cpu(struct perf_ftrace *ftrace)
 	return set_tracing_cpumask(cpumap);
 }
 
+static int set_tracing_func_stack_trace(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->func_stack_trace)
+		return 0;
+
+	if (write_tracing_option_file("func_stack_trace", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void)
 {
 	struct perf_cpu_map *cpumap = perf_cpu_map__new(NULL);
@@ -326,6 +357,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_func_stack_trace(ftrace) < 0) {
+		pr_err("failed to set tracing option func_stack_trace\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_filters(ftrace) < 0) {
 		pr_err("failed to set tracing filters\n");
 		goto out_reset;
@@ -459,6 +495,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "trace given functions only", parse_filter_func),
 	OPT_CALLBACK('N', "notrace-funcs", &ftrace.notrace, "func",
 		     "do not trace given functions", parse_filter_func),
+	OPT_BOOLEAN('s', "func-stack-trace", &ftrace.func_stack_trace,
+		    "Show kernel stack trace for function tracer"),
 	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
 		     "Set graph filter on given functions (imply to use function_graph tracer)",
 		     parse_filter_func, "*"),
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#11 lore raw

PATCH 5/19 perf ftrace: add option '-l/--list-functions' to list available functions

In reply to Changbin Du

This adds an option '-l/--list-functions' to list all available functions which is read from tracing file 'available_filter_functions'.

---
 tools/perf/builtin-ftrace.c | 43 +++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
tools/perf/builtin-ftrace.c +43 -0
index 1d30c2d5f88b..8133d910d5d8 100644
@@ -33,6 +33,7 @@ struct perf_ftrace {
 	struct evlist		*evlist;
 	struct target		target;
 	const char		*tracer;
+	bool			list_avail_functions;
 	struct list_head	filters;
 	struct list_head	notrace;
 	struct list_head	graph_funcs;
@@ -142,6 +143,43 @@ static int write_tracing_option_file(const char *name, const char *val)
 	return ret;
 }
 
+static int read_tracing_file_to_stdout(const char *name)
+{
+	char buf[4096];
+	char *file;
+	int fd;
+	int ret = -1;
+
+	file = get_tracing_file(name);
+	if (!file) {
+		pr_debug("cannot get tracing file: %s\n", name);
+		return -1;
+	}
+
+	fd = open(file, O_RDONLY);
+	if (fd < 0) {
+		pr_debug("cannot open tracing file: %s: %s\n",
+			 name, str_error_r(errno, buf, sizeof(buf)));
+		goto out;
+	}
+
+	/* read contents to stdout */
+	while (true) {
+		int n = read(fd, buf, sizeof(buf));
+		if (n <= 0)
+			goto out_close;
+		if (fwrite(buf, n, 1, stdout) != 1)
+			goto out_close;
+	}
+	ret = 0;
+
+out_close:
+	close(fd);
+out:
+	put_tracing_file(file);
+	return ret;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -332,6 +370,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
 
+	if (ftrace->list_avail_functions)
+		return read_tracing_file_to_stdout("available_filter_functions");
+
 	if (reset_tracing_files(ftrace) < 0) {
 		pr_err("failed to reset ftrace\n");
 		goto out;
@@ -483,6 +524,8 @@ int cmd_ftrace(int argc, const char **argv)
 		NULL
 	};
 	const struct option ftrace_options[] = {
+	OPT_BOOLEAN('L', "list-functions", &ftrace.list_avail_functions,
+		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
 	OPT_INCR('v', "verbose", &verbose,
Signature · 1 line
2.25.1
#12 lore raw

Re: [PATCH 05/19] perf ftrace: add option '-l/--list-functions' to list available functions

Em Sun, May 10, 2020 at 11:06:14PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-l/--list-functions' to list all available functions which is read from tracing file 'available_filter_functions'.

Here, in 'perf probe' we have:

[acme@five perf]$ perf probe -F tcp_* | head
tcp_abort
tcp_ack
tcp_ack_update_rtt.isra.0
tcp_add_backlog
tcp_add_reno_sack
tcp_adjust_pcount
tcp_alloc_md5sig_pool
tcp_any_retrans_done.part.0
tcp_assign_congestion_control
tcp_bpf_clone
[acme@five perf]$ perf probe -h -F

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -F, --funcs <[FILTER]>
                          Show potential probe-able functions.

[acme@five perf]$

So I think we should align with this.

- Arnaldo

Quoted 83 lines
---
 tools/perf/builtin-ftrace.c | 43 +++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
tools/perf/builtin-ftrace.c +43 -0
index 1d30c2d5f88b..8133d910d5d8 100644
@@ -33,6 +33,7 @@ struct perf_ftrace {
 	struct evlist		*evlist;
 	struct target		target;
 	const char		*tracer;
+	bool			list_avail_functions;
 	struct list_head	filters;
 	struct list_head	notrace;
 	struct list_head	graph_funcs;
@@ -142,6 +143,43 @@ static int write_tracing_option_file(const char *name, const char *val)
 	return ret;
 }
 
+static int read_tracing_file_to_stdout(const char *name)
+{
+	char buf[4096];
+	char *file;
+	int fd;
+	int ret = -1;
+
+	file = get_tracing_file(name);
+	if (!file) {
+		pr_debug("cannot get tracing file: %s\n", name);
+		return -1;
+	}
+
+	fd = open(file, O_RDONLY);
+	if (fd < 0) {
+		pr_debug("cannot open tracing file: %s: %s\n",
+			 name, str_error_r(errno, buf, sizeof(buf)));
+		goto out;
+	}
+
+	/* read contents to stdout */
+	while (true) {
+		int n = read(fd, buf, sizeof(buf));
+		if (n <= 0)
+			goto out_close;
+		if (fwrite(buf, n, 1, stdout) != 1)
+			goto out_close;
+	}
+	ret = 0;
+
+out_close:
+	close(fd);
+out:
+	put_tracing_file(file);
+	return ret;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -332,6 +370,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
 
+	if (ftrace->list_avail_functions)
+		return read_tracing_file_to_stdout("available_filter_functions");
+
 	if (reset_tracing_files(ftrace) < 0) {
 		pr_err("failed to reset ftrace\n");
 		goto out;
@@ -483,6 +524,8 @@ int cmd_ftrace(int argc, const char **argv)
 		NULL
 	};
 	const struct option ftrace_options[] = {
+	OPT_BOOLEAN('L', "list-functions", &ftrace.list_avail_functions,
+		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
 	OPT_INCR('v', "verbose", &verbose,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#13 lore raw

Re: [PATCH 05/19] perf ftrace: add option '-l/--list-functions' to list available functions

Quoted 30 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:14PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-l/--list-functions' to list all available functions which is read from tracing file 'available_filter_functions'.

Here, in 'perf probe' we have:

[acme@five perf]$ perf probe -F tcp_* | head
tcp_abort
tcp_ack
tcp_ack_update_rtt.isra.0
tcp_add_backlog
tcp_add_reno_sack
tcp_adjust_pcount
tcp_alloc_md5sig_pool
tcp_any_retrans_done.part.0
tcp_assign_congestion_control
tcp_bpf_clone
[acme@five perf]$ perf probe -h -F

Usage: perf probe [<options>] 'PROBEDEF' ['PROBEDEF' ...]

    or: perf probe [<options>] --add 'PROBEDEF' [--add 'PROBEDEF' ...]
    or: perf probe [<options>] --del '[GROUP:]EVENT' ...
    or: perf probe --list [GROUP:]EVENT ...
    or: perf probe [<options>] --line 'LINEDESC'
    or: perf probe [<options>] --vars 'PROBEPOINT'
    or: perf probe [<options>] --funcs

    -F, --funcs <[FILTER]>
                          Show potential probe-able functions.
Quoted 6 lines

[acme@five perf]$

So I think we should align with this.

- Arnaldo

Agreed. Will do that in next revision.

Quoted 87 lines
Quoted 83 lines
---
 tools/perf/builtin-ftrace.c | 43 +++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)
tools/perf/builtin-ftrace.c +43 -0
index 1d30c2d5f88b..8133d910d5d8 100644
@@ -33,6 +33,7 @@ struct perf_ftrace {
 	struct evlist		*evlist;
 	struct target		target;
 	const char		*tracer;
+	bool			list_avail_functions;
 	struct list_head	filters;
 	struct list_head	notrace;
 	struct list_head	graph_funcs;
@@ -142,6 +143,43 @@ static int write_tracing_option_file(const char *name, const char *val)
 	return ret;
 }
 
+static int read_tracing_file_to_stdout(const char *name)
+{
+	char buf[4096];
+	char *file;
+	int fd;
+	int ret = -1;
+
+	file = get_tracing_file(name);
+	if (!file) {
+		pr_debug("cannot get tracing file: %s\n", name);
+		return -1;
+	}
+
+	fd = open(file, O_RDONLY);
+	if (fd < 0) {
+		pr_debug("cannot open tracing file: %s: %s\n",
+			 name, str_error_r(errno, buf, sizeof(buf)));
+		goto out;
+	}
+
+	/* read contents to stdout */
+	while (true) {
+		int n = read(fd, buf, sizeof(buf));
+		if (n <= 0)
+			goto out_close;
+		if (fwrite(buf, n, 1, stdout) != 1)
+			goto out_close;
+	}
+	ret = 0;
+
+out_close:
+	close(fd);
+out:
+	put_tracing_file(file);
+	return ret;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -332,6 +370,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
 
+	if (ftrace->list_avail_functions)
+		return read_tracing_file_to_stdout("available_filter_functions");
+
 	if (reset_tracing_files(ftrace) < 0) {
 		pr_err("failed to reset ftrace\n");
 		goto out;
@@ -483,6 +524,8 @@ int cmd_ftrace(int argc, const char **argv)
 		NULL
 	};
 	const struct option ftrace_options[] = {
+	OPT_BOOLEAN('L', "list-functions", &ftrace.list_avail_functions,
+		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
 	OPT_INCR('v', "verbose", &verbose,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#14 lore raw

PATCH 6/19 perf ftrace: add support for trace option sleep-time

In reply to Changbin Du

This adds an option '--nosleep-time' which allow us only to measure on-CPU time. This option is function_graph tracer only.

---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 8133d910d5d8..d3fcf3b0b792 100644
@@ -40,6 +40,7 @@ struct perf_ftrace {
 	struct list_head	nograph_funcs;
 	int			graph_depth;
 	bool			func_stack_trace;
+	bool			nosleep_time;
 };
 
 struct filter_entry {
@@ -186,6 +187,7 @@ static void reset_tracing_filters(void);
 static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 {
 	write_tracing_option_file("func_stack_trace", "0");
+	write_tracing_option_file("sleep-time", "1");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -345,6 +347,17 @@ static int set_tracing_depth(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_sleep_time(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->nosleep_time)
+		return 0;
+
+	if (write_tracing_option_file("sleep-time", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -413,6 +426,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_sleep_time(ftrace) < 0) {
+		pr_err("failed to set tracing option sleep-time\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -548,6 +566,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     parse_filter_func),
 	OPT_INTEGER('D', "graph-depth", &ftrace.graph_depth,
 		    "Max depth for function graph tracer"),
+	OPT_BOOLEAN(0, "nosleep-time", &ftrace.nosleep_time,
+		    "Measure on-CPU time only (function_graph only)"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#15 lore raw

Re: [PATCH 06/19] perf ftrace: add support for trace option sleep-time

Em Sun, May 10, 2020 at 11:06:15PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--nosleep-time' which allow us only to measure on-CPU time. This option is function_graph tracer only.

This seems, for now, very specific to the function_graph tracer, so perhaps we should have a:

--function_graph_opts nosleep-time,other,another,etc

?

In time options that have equivalent in other perf tools can be promoted?

- Arnaldo

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 8133d910d5d8..d3fcf3b0b792 100644
@@ -40,6 +40,7 @@ struct perf_ftrace {
 	struct list_head	nograph_funcs;
 	int			graph_depth;
 	bool			func_stack_trace;
+	bool			nosleep_time;
 };
 
 struct filter_entry {
@@ -186,6 +187,7 @@ static void reset_tracing_filters(void);
 static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 {
 	write_tracing_option_file("func_stack_trace", "0");
+	write_tracing_option_file("sleep-time", "1");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -345,6 +347,17 @@ static int set_tracing_depth(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_sleep_time(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->nosleep_time)
+		return 0;
+
+	if (write_tracing_option_file("sleep-time", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -413,6 +426,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_sleep_time(ftrace) < 0) {
+		pr_err("failed to set tracing option sleep-time\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -548,6 +566,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     parse_filter_func),
 	OPT_INTEGER('D', "graph-depth", &ftrace.graph_depth,
 		    "Max depth for function graph tracer"),
+	OPT_BOOLEAN(0, "nosleep-time", &ftrace.nosleep_time,
+		    "Measure on-CPU time only (function_graph only)"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
maintainer
#16 lore raw

Re: [PATCH 06/19] perf ftrace: add support for trace option sleep-time

Quoted 11 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:15PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--nosleep-time' which allow us only to measure on-CPU time. This option is function_graph tracer only.

This seems, for now, very specific to the function_graph tracer, so perhaps we should have a:

--function_graph_opts nosleep-time,other,another,etc

?

Agreed. Also I don't want to add an option in a negative form as it's confusing. Actually, our option parser can recognize --no-xxx form automatically so adding a positive option (--xxx) can handle that too.

Thanks
Namhyung

#17 lore raw

Re: [PATCH 06/19] perf ftrace: add support for trace option sleep-time

Quoted 10 lines · Namhyung Kim
Quoted 8 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:15PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--nosleep-time' which allow us only to measure on-CPU time. This option is function_graph tracer only.

This seems, for now, very specific to the function_graph tracer, so perhaps we should have a:

Quoted 9 lines
Quoted 3 lines

--function_graph_opts nosleep-time,other,another,etc

?

Agreed. Also I don't want to add an option in a negative form as it's confusing. Actually, our option parser can recognize --no-xxx form automatically so adding a positive option (--xxx) can handle that too.

Agree. And for the name, maybe we can align to original trace options.

root@WRT-WX9:/sys/kernel/debug/tracing# cat trace_options
print-parent
nosym-offset
nosym-addr
noverbose
noraw
nohex
nobin
noblock
trace_printk
annotate
nouserstacktrace
nosym-userobj
noprintk-msg-only
context-info
nolatency-format
record-cmd
norecord-tgid
overwrite
nodisable_on_free
irq-info
markers
noevent-fork
function-trace
nofunction-fork
nodisplay-graph
nostacktrace
notest_nop_accept
notest_nop_refuse
Quoted 2 lines

Thanks
Namhyung

Signature · 2 lines
Cheers,
Changbin Du
#18 lore raw

PATCH 7/19 perf ftrace: add support for trace option funcgraph-irqs

In reply to Changbin Du

This adds an option '--nofuncgraph-irqs' to filter out functions executed in irq context.

---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index d3fcf3b0b792..b16600a16efa 100644
@@ -41,6 +41,7 @@ struct perf_ftrace {
 	int			graph_depth;
 	bool			func_stack_trace;
 	bool			nosleep_time;
+	bool			nofuncgraph_irqs;
 };
 
 struct filter_entry {
@@ -188,6 +189,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 {
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
+	write_tracing_option_file("funcgraph-irqs", "1");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -358,6 +360,17 @@ static int set_tracing_sleep_time(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->nofuncgraph_irqs)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-irqs", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -431,6 +444,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_irqs(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-irqs\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -568,6 +586,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Max depth for function graph tracer"),
 	OPT_BOOLEAN(0, "nosleep-time", &ftrace.nosleep_time,
 		    "Measure on-CPU time only (function_graph only)"),
+	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
+		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#19 lore raw

Re: [PATCH 07/19] perf ftrace: add support for trace option funcgraph-irqs

Em Sun, May 10, 2020 at 11:06:16PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--nofuncgraph-irqs' to filter out functions executed in irq context.

Ditto

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index d3fcf3b0b792..b16600a16efa 100644
@@ -41,6 +41,7 @@ struct perf_ftrace {
 	int			graph_depth;
 	bool			func_stack_trace;
 	bool			nosleep_time;
+	bool			nofuncgraph_irqs;
 };
 
 struct filter_entry {
@@ -188,6 +189,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 {
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
+	write_tracing_option_file("funcgraph-irqs", "1");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -358,6 +360,17 @@ static int set_tracing_sleep_time(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->nofuncgraph_irqs)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-irqs", "0") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -431,6 +444,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_irqs(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-irqs\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -568,6 +586,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Max depth for function graph tracer"),
 	OPT_BOOLEAN(0, "nosleep-time", &ftrace.nosleep_time,
 		    "Measure on-CPU time only (function_graph only)"),
+	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
+		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#20 lore raw

PATCH 8/19 perf ftrace: add option -l/--long-info to show more info

In reply to Changbin Du

Sometimes we want ftrace display more and longer information about trace.

$ sudo perf ftrace -G -l
 6800.190937 |   4)   <...>-7683   |   2.072 us    |  mutex_unlock();
 6800.190941 |   4)   <...>-7683   |   2.171 us    |  __fsnotify_parent();
 6800.190943 |   4)   <...>-7683   |   1.497 us    |  fsnotify();
 6800.190944 |   4)   <...>-7683   |   0.775 us    |  __sb_end_write();
 6800.190945 |   4)   <...>-7683   |   0.854 us    |  fpregs_assert_state_consistent();
 6800.190947 |   4)   <...>-7683   |               |  do_syscall_64() {
 6800.190948 |   4)   <...>-7683   |               |    __x64_sys_close() {
 6800.190948 |   4)   <...>-7683   |               |      __close_fd() {
 6800.190948 |   4)   <...>-7683   |   0.322 us    |        _raw_spin_lock();
 6800.190949 |   4)   <...>-7683   |               |        filp_close() {
 6800.190949 |   4)   <...>-7683   |   0.320 us    |          dnotify_flush();
 6800.190950 |   4)   <...>-7683   |   0.325 us    |          locks_remove_posix();
---
 tools/perf/builtin-ftrace.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
tools/perf/builtin-ftrace.c +28 -0
index b16600a16efa..f11f2d3431b0 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			long_info;
 };
 
 struct filter_entry {
@@ -190,6 +191,9 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-proc", "0");
+	write_tracing_option_file("funcgraph-abstime", "0");
+	write_tracing_option_file("irq-info", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -371,6 +375,23 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_long_info(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->long_info)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-proc", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("funcgraph-abstime", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("irq-info", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -449,6 +470,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_long_info(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime/irq-info\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -588,6 +614,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
+		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#21 lore raw

Re: [PATCH 08/19] perf ftrace: add option -l/--long-info to show more info

Em Sun, May 10, 2020 at 11:06:17PM +0800, Changbin Du escreveu:

Quoted 1 line

Sometimes we want ftrace display more and longer information about trace.

Humm, -v? Or that would bring too much stuff from other parts of perf? I guess so, perhaps as an option to the function-graph tracer, one that combines, as you do, several options provided by that tracer?

- Arnaldo

Quoted 89 lines
$ sudo perf ftrace -G -l
 6800.190937 |   4)   <...>-7683   |   2.072 us    |  mutex_unlock();
 6800.190941 |   4)   <...>-7683   |   2.171 us    |  __fsnotify_parent();
 6800.190943 |   4)   <...>-7683   |   1.497 us    |  fsnotify();
 6800.190944 |   4)   <...>-7683   |   0.775 us    |  __sb_end_write();
 6800.190945 |   4)   <...>-7683   |   0.854 us    |  fpregs_assert_state_consistent();
 6800.190947 |   4)   <...>-7683   |               |  do_syscall_64() {
 6800.190948 |   4)   <...>-7683   |               |    __x64_sys_close() {
 6800.190948 |   4)   <...>-7683   |               |      __close_fd() {
 6800.190948 |   4)   <...>-7683   |   0.322 us    |        _raw_spin_lock();
 6800.190949 |   4)   <...>-7683   |               |        filp_close() {
 6800.190949 |   4)   <...>-7683   |   0.320 us    |          dnotify_flush();
 6800.190950 |   4)   <...>-7683   |   0.325 us    |          locks_remove_posix();
---
 tools/perf/builtin-ftrace.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
tools/perf/builtin-ftrace.c +28 -0
index b16600a16efa..f11f2d3431b0 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			long_info;
 };
 
 struct filter_entry {
@@ -190,6 +191,9 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-proc", "0");
+	write_tracing_option_file("funcgraph-abstime", "0");
+	write_tracing_option_file("irq-info", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -371,6 +375,23 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_long_info(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->long_info)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-proc", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("funcgraph-abstime", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("irq-info", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -449,6 +470,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_long_info(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime/irq-info\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -588,6 +614,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
+		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
maintainer
#22 lore raw

Re: [PATCH 08/19] perf ftrace: add option -l/--long-info to show more info

Quoted 7 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:17PM +0800, Changbin Du escreveu:

Quoted 1 line

Sometimes we want ftrace display more and longer information about trace.

Humm, -v? Or that would bring too much stuff from other parts of perf? I guess so, perhaps as an option to the function-graph tracer, one that combines, as you do, several options provided by that tracer?

I think -v option is to debug perf tool's behavior while this is to change the output (or behavior).

Thanks,
Namhyung

#23 lore raw

Re: [PATCH 08/19] perf ftrace: add option -l/--long-info to show more info

In reply to Arnaldo Carvalho de Melo
Quoted 7 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:17PM +0800, Changbin Du escreveu:

Quoted 1 line

Sometimes we want ftrace display more and longer information about trace.

Humm, -v? Or that would bring too much stuff from other parts of perf? I guess so, perhaps as an option to the function-graph tracer, one that combines, as you do, several options provided by that tracer?

yes, this option enables mutiple trace options onetime, which will make the function tracer or graph tracer output as much as more information per-line. So I call it as 'long'.

Quoted 95 lines

- Arnaldo

Quoted 89 lines
$ sudo perf ftrace -G -l
 6800.190937 |   4)   <...>-7683   |   2.072 us    |  mutex_unlock();
 6800.190941 |   4)   <...>-7683   |   2.171 us    |  __fsnotify_parent();
 6800.190943 |   4)   <...>-7683   |   1.497 us    |  fsnotify();
 6800.190944 |   4)   <...>-7683   |   0.775 us    |  __sb_end_write();
 6800.190945 |   4)   <...>-7683   |   0.854 us    |  fpregs_assert_state_consistent();
 6800.190947 |   4)   <...>-7683   |               |  do_syscall_64() {
 6800.190948 |   4)   <...>-7683   |               |    __x64_sys_close() {
 6800.190948 |   4)   <...>-7683   |               |      __close_fd() {
 6800.190948 |   4)   <...>-7683   |   0.322 us    |        _raw_spin_lock();
 6800.190949 |   4)   <...>-7683   |               |        filp_close() {
 6800.190949 |   4)   <...>-7683   |   0.320 us    |          dnotify_flush();
 6800.190950 |   4)   <...>-7683   |   0.325 us    |          locks_remove_posix();
---
 tools/perf/builtin-ftrace.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
tools/perf/builtin-ftrace.c +28 -0
index b16600a16efa..f11f2d3431b0 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			long_info;
 };
 
 struct filter_entry {
@@ -190,6 +191,9 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-proc", "0");
+	write_tracing_option_file("funcgraph-abstime", "0");
+	write_tracing_option_file("irq-info", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -371,6 +375,23 @@ static int set_tracing_funcgraph_irqs(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_long_info(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->long_info)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-proc", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("funcgraph-abstime", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("irq-info", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -449,6 +470,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_long_info(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-proc/funcgraph-abstime/irq-info\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -588,6 +614,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
+		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#24 lore raw

PATCH 9/19 perf ftrace: add support for trace option tracing_thresh

In reply to Changbin Du

This adds an option '--tracing-thresh' to setup trace duration threshold for funcgraph tracer.

---
 tools/perf/builtin-ftrace.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
tools/perf/builtin-ftrace.c +26 -0
index f11f2d3431b0..20bc14d6c5fb 100644
@@ -43,6 +43,7 @@ struct perf_ftrace {
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			long_info;
+	unsigned		tracing_thresh;
 };
 
 struct filter_entry {
@@ -213,6 +214,9 @@ static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 	if (write_tracing_file("max_graph_depth", "0") < 0)
 		return -1;
 
+	if (write_tracing_file("tracing_thresh", "0") < 0)
+		return -1;
+
 	reset_tracing_filters();
 	reset_tracing_options(ftrace);
 	return 0;
@@ -392,6 +396,21 @@ static int set_tracing_long_info(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_thresh(struct perf_ftrace *ftrace)
+{
+	char buf[16];
+
+	if (ftrace->tracing_thresh == 0)
+		return 0;
+
+	snprintf(buf, sizeof(buf), "%d", ftrace->tracing_thresh);
+
+	if (write_tracing_file("tracing_thresh", buf) < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -475,6 +494,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_thresh(ftrace) < 0) {
+		pr_err("failed to set tracing thresh\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -616,6 +640,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
+	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
+		     "Only show functions of which the duration is greater than <n>µs"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#25 lore raw

Re: [PATCH 09/19] perf ftrace: add support for trace option tracing_thresh

Em Sun, May 10, 2020 at 11:06:18PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--tracing-thresh' to setup trace duration threshold for funcgraph tracer.

Ditto

Quoted 73 lines
---
 tools/perf/builtin-ftrace.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)
tools/perf/builtin-ftrace.c +26 -0
index f11f2d3431b0..20bc14d6c5fb 100644
@@ -43,6 +43,7 @@ struct perf_ftrace {
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			long_info;
+	unsigned		tracing_thresh;
 };
 
 struct filter_entry {
@@ -213,6 +214,9 @@ static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
 	if (write_tracing_file("max_graph_depth", "0") < 0)
 		return -1;
 
+	if (write_tracing_file("tracing_thresh", "0") < 0)
+		return -1;
+
 	reset_tracing_filters();
 	reset_tracing_options(ftrace);
 	return 0;
@@ -392,6 +396,21 @@ static int set_tracing_long_info(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_thresh(struct perf_ftrace *ftrace)
+{
+	char buf[16];
+
+	if (ftrace->tracing_thresh == 0)
+		return 0;
+
+	snprintf(buf, sizeof(buf), "%d", ftrace->tracing_thresh);
+
+	if (write_tracing_file("tracing_thresh", buf) < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -475,6 +494,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_thresh(ftrace) < 0) {
+		pr_err("failed to set tracing thresh\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -616,6 +640,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
+	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
+		     "Only show functions of which the duration is greater than <n>µs"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#26 lore raw

PATCH 10/19 perf ftrace: add support for trace option funcgraph-tail

In reply to Changbin Du

This adds an option '--funcgraph-tail' for function graph tracer.

---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 20bc14d6c5fb..2ef5d1c4b23c 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
 };
@@ -192,6 +193,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-tail", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -411,6 +413,17 @@ static int set_tracing_thresh(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->funcgraph_tail)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-tail", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -499,6 +512,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_tail(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-tail\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -638,6 +656,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
+		    "Show function tails comment (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
#27 lore raw

Re: [PATCH 10/19] perf ftrace: add support for trace option funcgraph-tail

Em Sun, May 10, 2020 at 11:06:19PM +0800, Changbin Du escreveu:

Quoted 1 line

This adds an option '--funcgraph-tail' for function graph tracer.

Ditto

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 20bc14d6c5fb..2ef5d1c4b23c 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
 };
@@ -192,6 +193,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-tail", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -411,6 +413,17 @@ static int set_tracing_thresh(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->funcgraph_tail)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-tail", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -499,6 +512,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_tail(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-tail\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -638,6 +656,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
+		    "Show function tails comment (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#28 lore raw

Re: [PATCH 10/19] perf ftrace: add support for trace option funcgraph-tail

In reply to Changbin Du

Em Sun, May 10, 2020 at 11:06:19PM +0800, Changbin Du escreveu:

Quoted 1 line

This adds an option '--funcgraph-tail' for function graph tracer.

And I think we should make these available in a compact way, as Intel PT has, i.e. instead of doing something like:

--function-graph-options nosleep_time,noirqs,no_tail,long_info

We could have:

-G ns,ni,nt,li

To save on typing, or something like that.

- Arnaldo

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 20bc14d6c5fb..2ef5d1c4b23c 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
 };
@@ -192,6 +193,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-tail", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -411,6 +413,17 @@ static int set_tracing_thresh(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->funcgraph_tail)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-tail", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -499,6 +512,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_tail(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-tail\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -638,6 +656,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
+		    "Show function tails comment (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
maintainer
#29 lore raw

Re: [PATCH 10/19] perf ftrace: add support for trace option funcgraph-tail

Quoted 7 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:19PM +0800, Changbin Du escreveu:

Quoted 1 line

This adds an option '--funcgraph-tail' for function graph tracer.

And I think we should make these available in a compact way, as Intel PT has, i.e. instead of doing something like:

Quoted 7 lines

--function-graph-options nosleep_time,noirqs,no_tail,long_info

We could have:

   -G ns,ni,nt,li

To save on typing, or something like that.

Looks good. What about separate 'no' prefix from two letter alias?

st, nost, ir, noir, ...

Thanks
Namhyung

#30 lore raw

Re: [PATCH 10/19] perf ftrace: add support for trace option funcgraph-tail

In reply to Arnaldo Carvalho de Melo

Hi Arnaldo,
First appologize for so long dealy! Maybe you have forgot the context of this serias ;)

Quoted 6 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:19PM +0800, Changbin Du escreveu:

Quoted 1 line

This adds an option '--funcgraph-tail' for function graph tracer.

And I think we should make these available in a compact way, as Intel PT has, i.e. instead of doing something like:

Quoted 8 lines

--function-graph-options nosleep_time,noirqs,no_tail,long_info

We could have:

   -G ns,ni,nt,li

To save on typing, or something like that.

I just noticed that there is an existing graph tracer option:

	--graph-depth <n>

Then our new option would be:
--graph-options depth=<n>,nosleep_time,noirqs,no_tail,long_info

This makes it a little complex to parse such options (need to parse key-vale pairs).

So can we follow the existing style instead? I mean we can have below options:

	-D --graph-depth <n>
	-G --graph-funcs <func>

	--graph-nosleep-time
	--graph-noirqs
	--graph-notail
	--graph-longinfo

All graph tracer options are prefixed with '--graph-'.

Quoted 73 lines

- Arnaldo

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index 20bc14d6c5fb..2ef5d1c4b23c 100644
@@ -42,6 +42,7 @@ struct perf_ftrace {
 	bool			func_stack_trace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
+	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
 };
@@ -192,6 +193,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("func_stack_trace", "0");
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
+	write_tracing_option_file("funcgraph-tail", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -411,6 +413,17 @@ static int set_tracing_thresh(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->funcgraph_tail)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-tail", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -499,6 +512,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_funcgraph_tail(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-tail\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -638,6 +656,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Measure on-CPU time only (function_graph only)"),
 	OPT_BOOLEAN(0, "nofuncgraph-irqs", &ftrace.nofuncgraph_irqs,
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
+	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
+		    "Show function tails comment (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#31 lore raw

PATCH 11/19 perf ftrace: add option '-u/--userstacktrace' to show userspace stacktrace

In reply to Changbin Du

This adds an option ''-u/--userstacktrace' for function tracer to display userspace back trace.

---
 tools/perf/builtin-ftrace.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tools/perf/builtin-ftrace.c +24 -0
index 2ef5d1c4b23c..ab76ba66bd9e 100644
@@ -40,6 +40,7 @@ struct perf_ftrace {
 	struct list_head	nograph_funcs;
 	int			graph_depth;
 	bool			func_stack_trace;
+	bool			userstacktrace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			funcgraph_tail;
@@ -197,6 +198,8 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
+	write_tracing_option_file("userstacktrace", "0");
+	write_tracing_option_file("sym-userobj", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -287,6 +290,20 @@ static int set_tracing_func_stack_trace(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_userstacktrace(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->userstacktrace)
+		return 0;
+
+	if (write_tracing_option_file("userstacktrace", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("sym-userobj", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void)
 {
 	struct perf_cpu_map *cpumap = perf_cpu_map__new(NULL);
@@ -482,6 +499,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_userstacktrace(ftrace) < 0) {
+		pr_err("failed to set tracing option userstacktrace\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_filters(ftrace) < 0) {
 		pr_err("failed to set tracing filters\n");
 		goto out_reset;
@@ -644,6 +666,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "do not trace given functions", parse_filter_func),
 	OPT_BOOLEAN('s', "func-stack-trace", &ftrace.func_stack_trace,
 		    "Show kernel stack trace for function tracer"),
+	OPT_BOOLEAN('u', "userstacktrace", &ftrace.userstacktrace,
+		    "Show stacktrace of the current user space thread"),
 	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
 		     "Set graph filter on given functions (imply to use function_graph tracer)",
 		     parse_filter_func, "*"),
Signature · 1 line
2.25.1
#32 lore raw

Re: [PATCH 11/19] perf ftrace: add option '-u/--userstacktrace' to show userspace stacktrace

Em Sun, May 10, 2020 at 11:06:20PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option ''-u/--userstacktrace' for function tracer to display userspace back trace.

Probably we should have this as a term, an option to --call-graph?

For --call-graph the way to suppress this is to ask for the event to be in the kernel only, i.e. something like:

	perf record -e cycles:k --call-graph

So perhaps we should have something like:

	perf ftrace --call-graph 

With some default, possibly a bit different than the other perf tools, just including the kernel, and accepting:

	perf ftrace --call-graph
	perf ftrace --call-graph k
	perf ftrace --call-graph u
	perf ftrace --call-graph uk

- Arnaldo

Quoted 71 lines
---
 tools/perf/builtin-ftrace.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tools/perf/builtin-ftrace.c +24 -0
index 2ef5d1c4b23c..ab76ba66bd9e 100644
@@ -40,6 +40,7 @@ struct perf_ftrace {
 	struct list_head	nograph_funcs;
 	int			graph_depth;
 	bool			func_stack_trace;
+	bool			userstacktrace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			funcgraph_tail;
@@ -197,6 +198,8 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
+	write_tracing_option_file("userstacktrace", "0");
+	write_tracing_option_file("sym-userobj", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -287,6 +290,20 @@ static int set_tracing_func_stack_trace(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_userstacktrace(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->userstacktrace)
+		return 0;
+
+	if (write_tracing_option_file("userstacktrace", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("sym-userobj", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void)
 {
 	struct perf_cpu_map *cpumap = perf_cpu_map__new(NULL);
@@ -482,6 +499,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_userstacktrace(ftrace) < 0) {
+		pr_err("failed to set tracing option userstacktrace\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_filters(ftrace) < 0) {
 		pr_err("failed to set tracing filters\n");
 		goto out_reset;
@@ -644,6 +666,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "do not trace given functions", parse_filter_func),
 	OPT_BOOLEAN('s', "func-stack-trace", &ftrace.func_stack_trace,
 		    "Show kernel stack trace for function tracer"),
+	OPT_BOOLEAN('u', "userstacktrace", &ftrace.userstacktrace,
+		    "Show stacktrace of the current user space thread"),
 	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
 		     "Set graph filter on given functions (imply to use function_graph tracer)",
 		     parse_filter_func, "*"),
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#33 lore raw

Re: [PATCH 11/19] perf ftrace: add option '-u/--userstacktrace' to show userspace stacktrace

Quoted 18 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:20PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option ''-u/--userstacktrace' for function tracer to display userspace back trace.

Probably we should have this as a term, an option to --call-graph?

For --call-graph the way to suppress this is to ask for the event to be in the kernel only, i.e. something like:

	perf record -e cycles:k --call-graph

So perhaps we should have something like:

	perf ftrace --call-graph 

With some default, possibly a bit different than the other perf tools, just including the kernel, and accepting:

Quoted 7 lines
	perf ftrace --call-graph
	perf ftrace --call-graph k
	perf ftrace --call-graph u
	perf ftrace --call-graph uk

- Arnaldo

I just found userstacktrace is not supported by ftrace yet. It is not hard to add this feature in kernel and I have done it locally. So I will drop this option before it is upstreamed.

Quoted 75 lines
Quoted 71 lines
---
 tools/perf/builtin-ftrace.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
tools/perf/builtin-ftrace.c +24 -0
index 2ef5d1c4b23c..ab76ba66bd9e 100644
@@ -40,6 +40,7 @@ struct perf_ftrace {
 	struct list_head	nograph_funcs;
 	int			graph_depth;
 	bool			func_stack_trace;
+	bool			userstacktrace;
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			funcgraph_tail;
@@ -197,6 +198,8 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
+	write_tracing_option_file("userstacktrace", "0");
+	write_tracing_option_file("sym-userobj", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -287,6 +290,20 @@ static int set_tracing_func_stack_trace(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_userstacktrace(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->userstacktrace)
+		return 0;
+
+	if (write_tracing_option_file("userstacktrace", "1") < 0)
+		return -1;
+
+	if (write_tracing_option_file("sym-userobj", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void)
 {
 	struct perf_cpu_map *cpumap = perf_cpu_map__new(NULL);
@@ -482,6 +499,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_userstacktrace(ftrace) < 0) {
+		pr_err("failed to set tracing option userstacktrace\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_filters(ftrace) < 0) {
 		pr_err("failed to set tracing filters\n");
 		goto out_reset;
@@ -644,6 +666,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "do not trace given functions", parse_filter_func),
 	OPT_BOOLEAN('s', "func-stack-trace", &ftrace.func_stack_trace,
 		    "Show kernel stack trace for function tracer"),
+	OPT_BOOLEAN('u', "userstacktrace", &ftrace.userstacktrace,
+		    "Show stacktrace of the current user space thread"),
 	OPT_CALLBACK_DEFAULT('G', "graph-funcs", &ftrace.graph_funcs, "func",
 		     "Set graph filter on given functions (imply to use function_graph tracer)",
 		     parse_filter_func, "*"),
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#34 lore raw

PATCH 12/19 perf ftrace: add support for tracing children processes

In reply to Changbin Du

This adds an option '--trace-children' to allow us trace children processes spawned by our target.

---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index ab76ba66bd9e..8fd95c109fe8 100644
@@ -46,6 +46,7 @@ struct perf_ftrace {
 	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
+	bool			trace_children;
 };
 
 struct filter_entry {
@@ -200,6 +201,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("irq-info", "0");
 	write_tracing_option_file("userstacktrace", "0");
 	write_tracing_option_file("sym-userobj", "0");
+	write_tracing_option_file("function-fork", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -441,6 +443,17 @@ static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_trace_children(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->trace_children)
+		return 0;
+
+	if (write_tracing_option_file("function-fork", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -539,6 +552,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_trace_children(ftrace) < 0) {
+		pr_err("failed to set tracing option function-fork\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -686,6 +704,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
 		     "Only show functions of which the duration is greater than <n>µs"),
+	OPT_BOOLEAN(0, "trace-children", &ftrace.trace_children,
+		    "Trace children processes"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#35 lore raw

Re: [PATCH 12/19] perf ftrace: add support for tracing children processes

Em Sun, May 10, 2020 at 11:06:21PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '--trace-children' to allow us trace children processes spawned by our target.

--inherit

- Arnaldo

Quoted 67 lines
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index ab76ba66bd9e..8fd95c109fe8 100644
@@ -46,6 +46,7 @@ struct perf_ftrace {
 	bool			funcgraph_tail;
 	bool			long_info;
 	unsigned		tracing_thresh;
+	bool			trace_children;
 };
 
 struct filter_entry {
@@ -200,6 +201,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("irq-info", "0");
 	write_tracing_option_file("userstacktrace", "0");
 	write_tracing_option_file("sym-userobj", "0");
+	write_tracing_option_file("function-fork", "0");
 }
 
 static int reset_tracing_files(struct perf_ftrace *ftrace __maybe_unused)
@@ -441,6 +443,17 @@ static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_trace_children(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->trace_children)
+		return 0;
+
+	if (write_tracing_option_file("function-fork", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -539,6 +552,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_trace_children(ftrace) < 0) {
+		pr_err("failed to set tracing option function-fork\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -686,6 +704,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
 		     "Only show functions of which the duration is greater than <n>µs"),
+	OPT_BOOLEAN(0, "trace-children", &ftrace.trace_children,
+		    "Trace children processes"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#36 lore raw

PATCH 13/19 perf ftrace: add option '-b/--buffer-size' to set per-cpu buffer size

In reply to Changbin Du

This adds an option '-b/--buffer-size' to allow us set the size of per-cpu tracing buffer.

---
 tools/perf/builtin-ftrace.c | 45 ++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 10 deletions(-)
tools/perf/builtin-ftrace.c +35 -10
index 8fd95c109fe8..a93fbdac6aa4 100644
@@ -47,6 +47,7 @@ struct perf_ftrace {
 	bool			long_info;
 	unsigned		tracing_thresh;
 	bool			trace_children;
+	unsigned		buffer_size_kb;
 };
 
 struct filter_entry {
@@ -187,6 +188,17 @@ static int read_tracing_file_to_stdout(const char *name)
 	return ret;
 }
 
+static int write_tracing_file_int(const char *name, int value)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", value);
+	if (write_tracing_file(name, buf) < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -360,8 +372,6 @@ static void reset_tracing_filters(void)
 
 static int set_tracing_depth(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->graph_depth == 0)
 		return 0;
 
@@ -370,9 +380,7 @@ static int set_tracing_depth(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->graph_depth);
-
-	if (write_tracing_file("max_graph_depth", buf) < 0)
+	if (write_tracing_file_int("max_graph_depth", ftrace->graph_depth) < 0)
 		return -1;
 
 	return 0;
@@ -419,14 +427,10 @@ static int set_tracing_long_info(struct perf_ftrace *ftrace)
 
 static int set_tracing_thresh(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->tracing_thresh == 0)
 		return 0;
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->tracing_thresh);
-
-	if (write_tracing_file("tracing_thresh", buf) < 0)
+	if (write_tracing_file_int("tracing_thresh", ftrace->tracing_thresh) < 0)
 		return -1;
 
 	return 0;
@@ -454,6 +458,20 @@ static int set_tracing_trace_children(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_buffer_size_kb(struct perf_ftrace *ftrace)
+{
+	int ret;
+
+	if (ftrace->buffer_size_kb == 0)
+		return 0;
+
+	ret = write_tracing_file_int("buffer_size_kb", ftrace->buffer_size_kb);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -557,6 +575,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_buffer_size_kb(ftrace) < 0) {
+		pr_err("failed to set tracing per-cpu buffer size\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -706,6 +729,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Only show functions of which the duration is greater than <n>µs"),
 	OPT_BOOLEAN(0, "trace-children", &ftrace.trace_children,
 		    "Trace children processes"),
+	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
+		     "size of per cpu buffer in kb"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#37 lore raw

Re: [PATCH 13/19] perf ftrace: add option '-b/--buffer-size' to set per-cpu buffer size

Em Sun, May 10, 2020 at 11:06:22PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-b/--buffer-size' to allow us set the size of per-cpu tracing buffer.

-m

Quoted 116 lines
---
 tools/perf/builtin-ftrace.c | 45 ++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 10 deletions(-)
tools/perf/builtin-ftrace.c +35 -10
index 8fd95c109fe8..a93fbdac6aa4 100644
@@ -47,6 +47,7 @@ struct perf_ftrace {
 	bool			long_info;
 	unsigned		tracing_thresh;
 	bool			trace_children;
+	unsigned		buffer_size_kb;
 };
 
 struct filter_entry {
@@ -187,6 +188,17 @@ static int read_tracing_file_to_stdout(const char *name)
 	return ret;
 }
 
+static int write_tracing_file_int(const char *name, int value)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", value);
+	if (write_tracing_file(name, buf) < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -360,8 +372,6 @@ static void reset_tracing_filters(void)
 
 static int set_tracing_depth(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->graph_depth == 0)
 		return 0;
 
@@ -370,9 +380,7 @@ static int set_tracing_depth(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->graph_depth);
-
-	if (write_tracing_file("max_graph_depth", buf) < 0)
+	if (write_tracing_file_int("max_graph_depth", ftrace->graph_depth) < 0)
 		return -1;
 
 	return 0;
@@ -419,14 +427,10 @@ static int set_tracing_long_info(struct perf_ftrace *ftrace)
 
 static int set_tracing_thresh(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->tracing_thresh == 0)
 		return 0;
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->tracing_thresh);
-
-	if (write_tracing_file("tracing_thresh", buf) < 0)
+	if (write_tracing_file_int("tracing_thresh", ftrace->tracing_thresh) < 0)
 		return -1;
 
 	return 0;
@@ -454,6 +458,20 @@ static int set_tracing_trace_children(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_buffer_size_kb(struct perf_ftrace *ftrace)
+{
+	int ret;
+
+	if (ftrace->buffer_size_kb == 0)
+		return 0;
+
+	ret = write_tracing_file_int("buffer_size_kb", ftrace->buffer_size_kb);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -557,6 +575,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_buffer_size_kb(ftrace) < 0) {
+		pr_err("failed to set tracing per-cpu buffer size\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -706,6 +729,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Only show functions of which the duration is greater than <n>µs"),
 	OPT_BOOLEAN(0, "trace-children", &ftrace.trace_children,
 		    "Trace children processes"),
+	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
+		     "size of per cpu buffer in kb"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#38 lore raw

Re: [PATCH 13/19] perf ftrace: add option '-b/--buffer-size' to set per-cpu buffer size

Quoted 6 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:22PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-b/--buffer-size' to allow us set the size of per-cpu tracing buffer.

-m

sure.

Quoted 120 lines
Quoted 116 lines
---
 tools/perf/builtin-ftrace.c | 45 ++++++++++++++++++++++++++++---------
 1 file changed, 35 insertions(+), 10 deletions(-)
tools/perf/builtin-ftrace.c +35 -10
index 8fd95c109fe8..a93fbdac6aa4 100644
@@ -47,6 +47,7 @@ struct perf_ftrace {
 	bool			long_info;
 	unsigned		tracing_thresh;
 	bool			trace_children;
+	unsigned		buffer_size_kb;
 };
 
 struct filter_entry {
@@ -187,6 +188,17 @@ static int read_tracing_file_to_stdout(const char *name)
 	return ret;
 }
 
+static int write_tracing_file_int(const char *name, int value)
+{
+	char buf[16];
+
+	snprintf(buf, sizeof(buf), "%d", value);
+	if (write_tracing_file(name, buf) < 0)
+		return -1;
+
+	return 0;
+}
+
 static int reset_tracing_cpu(void);
 static void reset_tracing_filters(void);
 
@@ -360,8 +372,6 @@ static void reset_tracing_filters(void)
 
 static int set_tracing_depth(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->graph_depth == 0)
 		return 0;
 
@@ -370,9 +380,7 @@ static int set_tracing_depth(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->graph_depth);
-
-	if (write_tracing_file("max_graph_depth", buf) < 0)
+	if (write_tracing_file_int("max_graph_depth", ftrace->graph_depth) < 0)
 		return -1;
 
 	return 0;
@@ -419,14 +427,10 @@ static int set_tracing_long_info(struct perf_ftrace *ftrace)
 
 static int set_tracing_thresh(struct perf_ftrace *ftrace)
 {
-	char buf[16];
-
 	if (ftrace->tracing_thresh == 0)
 		return 0;
 
-	snprintf(buf, sizeof(buf), "%d", ftrace->tracing_thresh);
-
-	if (write_tracing_file("tracing_thresh", buf) < 0)
+	if (write_tracing_file_int("tracing_thresh", ftrace->tracing_thresh) < 0)
 		return -1;
 
 	return 0;
@@ -454,6 +458,20 @@ static int set_tracing_trace_children(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_buffer_size_kb(struct perf_ftrace *ftrace)
+{
+	int ret;
+
+	if (ftrace->buffer_size_kb == 0)
+		return 0;
+
+	ret = write_tracing_file_int("buffer_size_kb", ftrace->buffer_size_kb);
+	if (ret < 0)
+		return ret;
+
+	return 0;
+}
+
 static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 {
 	char *trace_file;
@@ -557,6 +575,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_buffer_size_kb(ftrace) < 0) {
+		pr_err("failed to set tracing per-cpu buffer size\n");
+		goto out_reset;
+	}
+
 	if (write_tracing_file("current_tracer", ftrace->tracer) < 0) {
 		pr_err("failed to set current_tracer to %s\n", ftrace->tracer);
 		goto out_reset;
@@ -706,6 +729,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "Only show functions of which the duration is greater than <n>µs"),
 	OPT_BOOLEAN(0, "trace-children", &ftrace.trace_children,
 		    "Trace children processes"),
+	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
+		     "size of per cpu buffer in kb"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#39 lore raw

PATCH 14/19 perf ftrace: add option -P/--no-pager to disable pager

In reply to Changbin Du

Sometimes we want perf displays trace immediately. So this adds an option '-P/--no-pager' to disable pager if needed.

---
 tools/perf/builtin-ftrace.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +14 -3
index a93fbdac6aa4..64c22f367ba2 100644
@@ -48,6 +48,7 @@ struct perf_ftrace {
 	unsigned		tracing_thresh;
 	bool			trace_children;
 	unsigned		buffer_size_kb;
+	bool			no_pager;
 };
 
 struct filter_entry {
@@ -56,6 +57,7 @@ struct filter_entry {
 };
 
 static volatile int workload_exec_errno;
+static bool interrupted;
 static bool done;
 
 static void sig_handler(int sig __maybe_unused)
@@ -63,6 +65,12 @@ static void sig_handler(int sig __maybe_unused)
 	done = true;
 }
 
+static void sig_handler_int(int sig __maybe_unused)
+{
+	sig_handler(sig);
+	interrupted = 1;
+}
+
 /*
  * perf_evlist__prepare_workload will send a SIGUSR1 if the fork fails, since
  * we asked by setting its exec_error to the function below,
@@ -492,7 +500,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		return -1;
 	}
 
-	signal(SIGINT, sig_handler);
+	signal(SIGINT, sig_handler_int);
 	signal(SIGUSR1, sig_handler);
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
@@ -585,7 +593,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
-	setup_pager();
+	if (!ftrace->no_pager)
+		setup_pager();
 
 	trace_file = get_tracing_file("trace_pipe");
 	if (!trace_file) {
@@ -636,7 +645,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
+	while (true && !interrupted) {
 		int n = read(trace_fd, buf, sizeof(buf));
 		if (n <= 0)
 			break;
@@ -731,6 +740,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Trace children processes"),
 	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
 		     "size of per cpu buffer in kb"),
+	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
+		    "Do not use pager"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#40 lore raw

Re: [PATCH 14/19] perf ftrace: add option -P/--no-pager to disable pager

Em Sun, May 10, 2020 at 11:06:23PM +0800, Changbin Du escreveu:

Quoted 2 lines

Sometimes we want perf displays trace immediately. So this adds an option '-P/--no-pager' to disable pager if needed.

Try:

perf record sleep 5
perf script

Then:

perf --no-pager script

I.e. its there already.

- Arnaldo

Quoted 79 lines
---
 tools/perf/builtin-ftrace.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +14 -3
index a93fbdac6aa4..64c22f367ba2 100644
@@ -48,6 +48,7 @@ struct perf_ftrace {
 	unsigned		tracing_thresh;
 	bool			trace_children;
 	unsigned		buffer_size_kb;
+	bool			no_pager;
 };
 
 struct filter_entry {
@@ -56,6 +57,7 @@ struct filter_entry {
 };
 
 static volatile int workload_exec_errno;
+static bool interrupted;
 static bool done;
 
 static void sig_handler(int sig __maybe_unused)
@@ -63,6 +65,12 @@ static void sig_handler(int sig __maybe_unused)
 	done = true;
 }
 
+static void sig_handler_int(int sig __maybe_unused)
+{
+	sig_handler(sig);
+	interrupted = 1;
+}
+
 /*
  * perf_evlist__prepare_workload will send a SIGUSR1 if the fork fails, since
  * we asked by setting its exec_error to the function below,
@@ -492,7 +500,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		return -1;
 	}
 
-	signal(SIGINT, sig_handler);
+	signal(SIGINT, sig_handler_int);
 	signal(SIGUSR1, sig_handler);
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
@@ -585,7 +593,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
-	setup_pager();
+	if (!ftrace->no_pager)
+		setup_pager();
 
 	trace_file = get_tracing_file("trace_pipe");
 	if (!trace_file) {
@@ -636,7 +645,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
+	while (true && !interrupted) {
 		int n = read(trace_fd, buf, sizeof(buf));
 		if (n <= 0)
 			break;
@@ -731,6 +740,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Trace children processes"),
 	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
 		     "size of per cpu buffer in kb"),
+	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
+		    "Do not use pager"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#41 lore raw

Re: [PATCH 14/19] perf ftrace: add option -P/--no-pager to disable pager

Quoted 10 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:23PM +0800, Changbin Du escreveu:

Quoted 2 lines

Sometimes we want perf displays trace immediately. So this adds an option '-P/--no-pager' to disable pager if needed.

Try:

perf record sleep 5
perf script

Quoted 8 lines

Then:

perf --no-pager script

I.e. its there already.

- Arnaldo

Yes, just tried this and it works:
perf --no-pager ftrace

So, the problem is that this option is not documented.

Quoted 83 lines
Quoted 79 lines
---
 tools/perf/builtin-ftrace.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +14 -3
index a93fbdac6aa4..64c22f367ba2 100644
@@ -48,6 +48,7 @@ struct perf_ftrace {
 	unsigned		tracing_thresh;
 	bool			trace_children;
 	unsigned		buffer_size_kb;
+	bool			no_pager;
 };
 
 struct filter_entry {
@@ -56,6 +57,7 @@ struct filter_entry {
 };
 
 static volatile int workload_exec_errno;
+static bool interrupted;
 static bool done;
 
 static void sig_handler(int sig __maybe_unused)
@@ -63,6 +65,12 @@ static void sig_handler(int sig __maybe_unused)
 	done = true;
 }
 
+static void sig_handler_int(int sig __maybe_unused)
+{
+	sig_handler(sig);
+	interrupted = 1;
+}
+
 /*
  * perf_evlist__prepare_workload will send a SIGUSR1 if the fork fails, since
  * we asked by setting its exec_error to the function below,
@@ -492,7 +500,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		return -1;
 	}
 
-	signal(SIGINT, sig_handler);
+	signal(SIGINT, sig_handler_int);
 	signal(SIGUSR1, sig_handler);
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
@@ -585,7 +593,8 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
-	setup_pager();
+	if (!ftrace->no_pager)
+		setup_pager();
 
 	trace_file = get_tracing_file("trace_pipe");
 	if (!trace_file) {
@@ -636,7 +645,7 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	}
 
 	/* read remaining buffer contents */
-	while (true) {
+	while (true && !interrupted) {
 		int n = read(trace_fd, buf, sizeof(buf));
 		if (n <= 0)
 			break;
@@ -731,6 +740,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Trace children processes"),
 	OPT_UINTEGER('b', "buffer-size", &ftrace.buffer_size_kb,
 		     "size of per cpu buffer in kb"),
+	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
+		    "Do not use pager"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#42 lore raw

PATCH 15/19 perf ftrace: show trace column header

In reply to Changbin Du

This makes perf-ftrace display column header before printing trace.

$ sudo perf ftrace
\# tracer: function
\#
\# entries-in-buffer/entries-written: 0/0   #P:8
\#
\#           TASK-PID     CPU#   TIMESTAMP  FUNCTION
\#              | |         |       |         |
           <...>-9246  [006]  10726.262760: mutex_unlock <-rb_simple_write
           <...>-9246  [006]  10726.262764: __fsnotify_parent <-vfs_write
           <...>-9246  [006]  10726.262765: fsnotify <-vfs_write
           <...>-9246  [006]  10726.262766: __sb_end_write <-vfs_write
           <...>-9246  [006]  10726.262767: fpregs_assert_state_consistent <-do_syscall_64
---
 tools/perf/builtin-ftrace.c | 3 +++
 1 file changed, 3 insertions(+)
tools/perf/builtin-ftrace.c +3 -0
index 64c22f367ba2..0b39b6a88026 100644
@@ -614,6 +614,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
 	pollfd.fd = trace_fd;
 
+	/* display column headers */
+	read_tracing_file_to_stdout("trace");
+
 	if (write_tracing_file("tracing_on", "1") < 0) {
 		pr_err("can't enable tracing\n");
 		goto out_close_fd;
Signature · 1 line
2.25.1
#43 lore raw

Re: [PATCH 15/19] perf ftrace: show trace column header

Em Sun, May 10, 2020 at 11:06:24PM +0800, Changbin Du escreveu:

Quoted 1 line

This makes perf-ftrace display column header before printing trace.

[acme@five perf]$ perf report -h header

Usage: perf report [<options>]

--header Show data header.
--header-only Show only data header.

[acme@five perf]$

Perhaps there is value in --header-only to show the default that will be setup when using some set of options.

- Arnaldo

Quoted 35 lines
$ sudo perf ftrace
\# tracer: function
\#
\# entries-in-buffer/entries-written: 0/0   #P:8
\#
\#           TASK-PID     CPU#   TIMESTAMP  FUNCTION
\#              | |         |       |         |
           <...>-9246  [006]  10726.262760: mutex_unlock <-rb_simple_write
           <...>-9246  [006]  10726.262764: __fsnotify_parent <-vfs_write
           <...>-9246  [006]  10726.262765: fsnotify <-vfs_write
           <...>-9246  [006]  10726.262766: __sb_end_write <-vfs_write
           <...>-9246  [006]  10726.262767: fpregs_assert_state_consistent <-do_syscall_64
---
 tools/perf/builtin-ftrace.c | 3 +++
 1 file changed, 3 insertions(+)
tools/perf/builtin-ftrace.c +3 -0
index 64c22f367ba2..0b39b6a88026 100644
@@ -614,6 +614,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
 	pollfd.fd = trace_fd;
 
+	/* display column headers */
+	read_tracing_file_to_stdout("trace");
+
 	if (write_tracing_file("tracing_on", "1") < 0) {
 		pr_err("can't enable tracing\n");
 		goto out_close_fd;
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#44 lore raw

Re: [PATCH 15/19] perf ftrace: show trace column header

Quoted 7 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:24PM +0800, Changbin Du escreveu:

Quoted 1 line

This makes perf-ftrace display column header before printing trace.

[acme@five perf]$ perf report -h header

Usage: perf report [<options>]

Quoted 8 lines

--header Show data header.
--header-only Show only data header.

[acme@five perf]$

Perhaps there is value in --header-only to show the default that will be setup when using some set of options.

Arnaldo, what is the '--header-only' option used for? As for ftrace, the header format is different for different trace options.

Quoted 41 lines

- Arnaldo

Quoted 35 lines
$ sudo perf ftrace
\# tracer: function
\#
\# entries-in-buffer/entries-written: 0/0   #P:8
\#
\#           TASK-PID     CPU#   TIMESTAMP  FUNCTION
\#              | |         |       |         |
           <...>-9246  [006]  10726.262760: mutex_unlock <-rb_simple_write
           <...>-9246  [006]  10726.262764: __fsnotify_parent <-vfs_write
           <...>-9246  [006]  10726.262765: fsnotify <-vfs_write
           <...>-9246  [006]  10726.262766: __sb_end_write <-vfs_write
           <...>-9246  [006]  10726.262767: fpregs_assert_state_consistent <-do_syscall_64
---
 tools/perf/builtin-ftrace.c | 3 +++
 1 file changed, 3 insertions(+)
tools/perf/builtin-ftrace.c +3 -0
index 64c22f367ba2..0b39b6a88026 100644
@@ -614,6 +614,9 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	fcntl(trace_fd, F_SETFL, O_NONBLOCK);
 	pollfd.fd = trace_fd;
 
+	/* display column headers */
+	read_tracing_file_to_stdout("trace");
+
 	if (write_tracing_file("tracing_on", "1") < 0) {
 		pr_err("can't enable tracing\n");
 		goto out_close_fd;
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#45 lore raw

PATCH 16/19 perf ftrace: add option -t/--tid to filter by thread id

In reply to Changbin Du

This allows us to trace single thread instead of the whole process.

---
 tools/perf/builtin-ftrace.c | 2 ++
 1 file changed, 2 insertions(+)
tools/perf/builtin-ftrace.c +2 -0
index 0b39b6a88026..8d04e5afe2d3 100644
@@ -707,6 +707,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
+	OPT_STRING('t', "tid", &ftrace.target.tid, "tid",
+		   "trace on existing thread id (exclusive to --pid)"),
 	OPT_INCR('v', "verbose", &verbose,
 		 "be more verbose"),
 	OPT_BOOLEAN('a', "all-cpus", &ftrace.target.system_wide,
Signature · 1 line
2.25.1
#46 lore raw

Re: [PATCH 16/19] perf ftrace: add option -t/--tid to filter by thread id

Em Sun, May 10, 2020 at 11:06:25PM +0800, Changbin Du escreveu:

Quoted 1 line

This allows us to trace single thread instead of the whole process.

I was going to adjust the patch to add, but you forgot to add the entry to tools/perf/Documentation/perf-ftrace.txt,

Please do so,

- Arnaldo

Quoted 21 lines
---
 tools/perf/builtin-ftrace.c | 2 ++
 1 file changed, 2 insertions(+)
tools/perf/builtin-ftrace.c +2 -0
index 0b39b6a88026..8d04e5afe2d3 100644
@@ -707,6 +707,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
+	OPT_STRING('t', "tid", &ftrace.target.tid, "tid",
+		   "trace on existing thread id (exclusive to --pid)"),
 	OPT_INCR('v', "verbose", &verbose,
 		 "be more verbose"),
 	OPT_BOOLEAN('a', "all-cpus", &ftrace.target.system_wide,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#47 lore raw

Re: [PATCH 16/19] perf ftrace: add option -t/--tid to filter by thread id

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:25PM +0800, Changbin Du escreveu:

Quoted 1 line

This allows us to trace single thread instead of the whole process.

I was going to adjust the patch to add, but you forgot to add the entry to tools/perf/Documentation/perf-ftrace.txt,

Please do so,

no problem. I will apend a new patch to update all the documentation.

Quoted 27 lines

- Arnaldo

Quoted 21 lines
---
 tools/perf/builtin-ftrace.c | 2 ++
 1 file changed, 2 insertions(+)
tools/perf/builtin-ftrace.c +2 -0
index 0b39b6a88026..8d04e5afe2d3 100644
@@ -707,6 +707,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "List available functions to filter"),
 	OPT_STRING('p', "pid", &ftrace.target.pid, "pid",
 		   "trace on existing process id"),
+	OPT_STRING('t', "tid", &ftrace.target.tid, "tid",
+		   "trace on existing thread id (exclusive to --pid)"),
 	OPT_INCR('v', "verbose", &verbose,
 		 "be more verbose"),
 	OPT_BOOLEAN('a', "all-cpus", &ftrace.target.system_wide,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#48 lore raw

PATCH 17/19 perf ftrace: add option -d/--delay to delay tracing

In reply to Changbin Du

This adds an option '-d/--delay' to allow us to start tracing some times later after workload is launched.

---
 tools/perf/builtin-ftrace.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +16 -3
index 8d04e5afe2d3..d376b37c53fc 100644
@@ -49,6 +49,7 @@ struct perf_ftrace {
 	bool			trace_children;
 	unsigned		buffer_size_kb;
 	bool			no_pager;
+	unsigned		initial_delay;
 };
 
 struct filter_entry {
@@ -617,13 +618,23 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	/* display column headers */
 	read_tracing_file_to_stdout("trace");
 
-	if (write_tracing_file("tracing_on", "1") < 0) {
-		pr_err("can't enable tracing\n");
-		goto out_close_fd;
+	if (!ftrace->initial_delay) {
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
 	}
 
 	perf_evlist__start_workload(ftrace->evlist);
 
+	if (ftrace->initial_delay) {
+		usleep(ftrace->initial_delay * 1000);
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
+	}
+
 	while (!done) {
 		if (poll(&pollfd, 1, -1) < 0)
 			break;
@@ -747,6 +758,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "size of per cpu buffer in kb"),
 	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
 		    "Do not use pager"),
+	OPT_UINTEGER('d', "delay", &ftrace.initial_delay,
+		     "Wait <n> ms before tracing"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
#49 lore raw

Re: [PATCH 17/19] perf ftrace: add option -d/--delay to delay tracing

Em Sun, May 10, 2020 at 11:06:26PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-d/--delay' to allow us to start tracing some times later after workload is launched.

[acme@five perf]$ perf record -h delay

Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-D, --delay <n> ms to wait before starting measurement after program start

[acme@five perf]$

Quoted 56 lines
---
 tools/perf/builtin-ftrace.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +16 -3
index 8d04e5afe2d3..d376b37c53fc 100644
@@ -49,6 +49,7 @@ struct perf_ftrace {
 	bool			trace_children;
 	unsigned		buffer_size_kb;
 	bool			no_pager;
+	unsigned		initial_delay;
 };
 
 struct filter_entry {
@@ -617,13 +618,23 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	/* display column headers */
 	read_tracing_file_to_stdout("trace");
 
-	if (write_tracing_file("tracing_on", "1") < 0) {
-		pr_err("can't enable tracing\n");
-		goto out_close_fd;
+	if (!ftrace->initial_delay) {
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
 	}
 
 	perf_evlist__start_workload(ftrace->evlist);
 
+	if (ftrace->initial_delay) {
+		usleep(ftrace->initial_delay * 1000);
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
+	}
+
 	while (!done) {
 		if (poll(&pollfd, 1, -1) < 0)
 			break;
@@ -747,6 +758,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "size of per cpu buffer in kb"),
 	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
 		    "Do not use pager"),
+	OPT_UINTEGER('d', "delay", &ftrace.initial_delay,
+		     "Wait <n> ms before tracing"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#50 lore raw

Re: [PATCH 17/19] perf ftrace: add option -d/--delay to delay tracing

Quoted 4 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:26PM +0800, Changbin Du escreveu:

Quoted 2 lines

This adds an option '-d/--delay' to allow us to start tracing some times later after workload is launched.

Quoted 9 lines

[acme@five perf]$ perf record -h delay

Usage: perf record [<options>] [<command>]
or: perf record [<options>] -- <command> [<options>]

-D, --delay <n> ms to wait before starting measurement after program start

[acme@five perf]$

Sure, will change the short option name.

Quoted 60 lines
Quoted 56 lines
---
 tools/perf/builtin-ftrace.c | 19 ++++++++++++++++---
 1 file changed, 16 insertions(+), 3 deletions(-)
tools/perf/builtin-ftrace.c +16 -3
index 8d04e5afe2d3..d376b37c53fc 100644
@@ -49,6 +49,7 @@ struct perf_ftrace {
 	bool			trace_children;
 	unsigned		buffer_size_kb;
 	bool			no_pager;
+	unsigned		initial_delay;
 };
 
 struct filter_entry {
@@ -617,13 +618,23 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 	/* display column headers */
 	read_tracing_file_to_stdout("trace");
 
-	if (write_tracing_file("tracing_on", "1") < 0) {
-		pr_err("can't enable tracing\n");
-		goto out_close_fd;
+	if (!ftrace->initial_delay) {
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
 	}
 
 	perf_evlist__start_workload(ftrace->evlist);
 
+	if (ftrace->initial_delay) {
+		usleep(ftrace->initial_delay * 1000);
+		if (write_tracing_file("tracing_on", "1") < 0) {
+			pr_err("can't enable tracing\n");
+			goto out_close_fd;
+		}
+	}
+
 	while (!done) {
 		if (poll(&pollfd, 1, -1) < 0)
 			break;
@@ -747,6 +758,8 @@ int cmd_ftrace(int argc, const char **argv)
 		     "size of per cpu buffer in kb"),
 	OPT_BOOLEAN('P', "no-pager", &ftrace.no_pager,
 		    "Do not use pager"),
+	OPT_UINTEGER('d', "delay", &ftrace.initial_delay,
+		     "Wait <n> ms before tracing"),
 	OPT_END()
 	};
 
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
Signature · 2 lines
Cheers,
Changbin Du
#51 lore raw

PATCH 18/19 perf ftrace: add option --latency-format to display more info about delay

In reply to Changbin Du

This is for the function graph tracer to display more info about latency. The execution context is shown in this case.

$ sudo perf ftrace -G --latency-format
\# tracer: function_graph
\#
 1)  .... |   0.992 us    |  mutex_unlock();
 1)  .... |   1.404 us    |  __fsnotify_parent();
 1)  .... |   1.023 us    |  fsnotify();
 1)  .... |   0.335 us    |  __sb_end_write();
 1)  d... |   0.439 us    |  fpregs_assert_state_consistent();
 1)  d... |               |  do_syscall_64() {
 1)  .... |               |    __x64_sys_close() {
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index d376b37c53fc..fd8e2f305136 100644
@@ -44,6 +44,7 @@ struct perf_ftrace {
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			funcgraph_tail;
+	bool			latency_format;
 	bool			long_info;
 	unsigned		tracing_thresh;
 	bool			trace_children;
@@ -217,6 +218,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
 	write_tracing_option_file("funcgraph-tail", "0");
+	write_tracing_option_file("latency-format", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -456,6 +458,17 @@ static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_latency_format(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->latency_format)
+		return 0;
+
+	if (write_tracing_option_file("latency-format", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_trace_children(struct perf_ftrace *ftrace)
 {
 	if (!ftrace->trace_children)
@@ -579,6 +592,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_latency_format(ftrace) < 0) {
+		pr_err("failed to set tracing option latency-format\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_trace_children(ftrace) < 0) {
 		pr_err("failed to set tracing option function-fork\n");
 		goto out_reset;
@@ -748,6 +766,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
 		    "Show function tails comment (function_graph only)"),
+	OPT_BOOLEAN(0, "latency-format", &ftrace.latency_format,
+		    "displays additional information about the latency (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
#52 lore raw

Re: [PATCH 18/19] perf ftrace: add option --latency-format to display more info about delay

Em Sun, May 10, 2020 at 11:06:27PM +0800, Changbin Du escreveu:

Quoted 2 lines

This is for the function graph tracer to display more info about latency. The execution context is shown in this case.

Ok, another --function-graph-opts entry, maybe you can just make -G receive optional args.

- Arnaldo

Quoted 78 lines
$ sudo perf ftrace -G --latency-format
\# tracer: function_graph
\#
 1)  .... |   0.992 us    |  mutex_unlock();
 1)  .... |   1.404 us    |  __fsnotify_parent();
 1)  .... |   1.023 us    |  fsnotify();
 1)  .... |   0.335 us    |  __sb_end_write();
 1)  d... |   0.439 us    |  fpregs_assert_state_consistent();
 1)  d... |               |  do_syscall_64() {
 1)  .... |               |    __x64_sys_close() {
---
 tools/perf/builtin-ftrace.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
tools/perf/builtin-ftrace.c +20 -0
index d376b37c53fc..fd8e2f305136 100644
@@ -44,6 +44,7 @@ struct perf_ftrace {
 	bool			nosleep_time;
 	bool			nofuncgraph_irqs;
 	bool			funcgraph_tail;
+	bool			latency_format;
 	bool			long_info;
 	unsigned		tracing_thresh;
 	bool			trace_children;
@@ -217,6 +218,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("sleep-time", "1");
 	write_tracing_option_file("funcgraph-irqs", "1");
 	write_tracing_option_file("funcgraph-tail", "0");
+	write_tracing_option_file("latency-format", "0");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
 	write_tracing_option_file("irq-info", "0");
@@ -456,6 +458,17 @@ static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_latency_format(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->latency_format)
+		return 0;
+
+	if (write_tracing_option_file("latency-format", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_trace_children(struct perf_ftrace *ftrace)
 {
 	if (!ftrace->trace_children)
@@ -579,6 +592,11 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace, int argc, const char **argv)
 		goto out_reset;
 	}
 
+	if (set_tracing_latency_format(ftrace) < 0) {
+		pr_err("failed to set tracing option latency-format\n");
+		goto out_reset;
+	}
+
 	if (set_tracing_trace_children(ftrace) < 0) {
 		pr_err("failed to set tracing option function-fork\n");
 		goto out_reset;
@@ -748,6 +766,8 @@ int cmd_ftrace(int argc, const char **argv)
 		    "Ignore functions that happen inside interrupt (function_graph only)"),
 	OPT_BOOLEAN(0, "funcgraph-tail", &ftrace.funcgraph_tail,
 		    "Show function tails comment (function_graph only)"),
+	OPT_BOOLEAN(0, "latency-format", &ftrace.latency_format,
+		    "displays additional information about the latency (function_graph only)"),
 	OPT_BOOLEAN('l', "long-info", &ftrace.long_info,
 		    "Show process names, PIDs, timestamps, irq-info if available"),
 	OPT_UINTEGER(0, "tracing-thresh", &ftrace.tracing_thresh,
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#53 lore raw

PATCH 19/19 perf ftrace: add change log

In reply to Changbin Du

Add a change log after previous enhancements.

---
 tools/perf/builtin-ftrace.c | 1 +
 1 file changed, 1 insertion(+)
tools/perf/builtin-ftrace.c +1 -0
index fd8e2f305136..8170746f94fd 100644
@@ -3,6 +3,7 @@
  * builtin-ftrace.c
  *
  * Copyright (c) 2013  LG Electronics,  Namhyung Kim <[email protected]>
+ * Copyright (c) 2020  Changbin Du <[email protected]>, significant enhancement.
  */
 
 #include "builtin.h"
Signature · 1 line
2.25.1
#54 lore raw
In reply to Changbin Du
Quoted 41 lines · On Sun, 10 May 2020 23:06:09 +0800 Changbin Du <[email protected]> wrote:

The perf has basic kernel ftrace support but lack support of most tracing options. This serias is target to enhance the perf ftrace functionality so that we can make full use of kernel ftrace with only perf.

In general, this serias be cataloged into two main changes:
1) Improve usability of existing functions. For example, we don't need to type extra option to select the tracer.
2) Add new options to support all other ftrace functions.

Here is a glance of all ftrace functions with this serias:
* - improved existing options.
+ - new added options.

$ sudo perf ftrace -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- <command> [<options>]

* -a, --all-cpus system-wide collection from all CPUs

(quoted diff) +2 -0
+   -b, --buffer-size <n>
                          size of per cpu buffer in kb
    -C, --cpu <cpu>       list of cpus to monitor
+   -d, --delay <n>       Wait <n> ms before tracing
    -D, --graph-depth <n>
                          Max depth for function graph tracer

* -G, --graph-funcs <func>
Set graph filter on given functions (imply to use function_graph tracer)
-g, --nograph-funcs <func>
Set nograph filter on given functions (imply to use function_graph tracer)

(quoted diff) +5 -0
+   -L, --list-functions  List available functions to filter
+   -l, --long-info       Show process names, PIDs, timestamps, irq-info if available
    -N, --notrace-funcs <func>
                          do not trace given functions
+   -P, --no-pager        Do not use pager
    -p, --pid <pid>       trace on existing process id
+   -s, --func-stack-trace
                          Show kernel stack trace for function tracer
+   -t, --tid <tid>       trace on existing thread id (exclusive to --pid)
    -T, --trace-funcs <func>
                          trace given functions only

Review

Quoted 10 lines
(quoted diff) +7 -0
+   -u, --userstacktrace  Show stacktrace of the current user space thread
    -v, --verbose         be more verbose
+       --funcgraph-tail  Show function tails comment (function_graph only)
+       --latency-format  displays additional information about the latency (function_graph only)
+       --nofuncgraph-irqs
                          Ignore functions that happen inside interrupt (function_graph only)
+       --nosleep-time    Measure on-CPU time only (function_graph only)
+       --trace-children  Trace children processes
+       --tracing-thresh <n>
                          Only show functions of which the duration is greater than <n>µs

Note, we are working on making more of the trace-cmd functionality into libraries. See this work here:

https://lore.kernel.org/r/[email protected]

Which introduces a libtracefs, that is to handle all the work needed to interact with the tracefs directory. This will also be useful for perf to read the event directory without having to open code that work.

I'm all for giving perf the functionality of ftrace, but I would like to have it do so with a more generic solution that other tools could benefit from as well.

Thanks!

-- Steve

#55 lore raw
Quoted 73 lines · Steven Rostedt
Quoted 1 line · On Sun, 10 May 2020 23:06:09 +0800 Changbin Du <[email protected]> wrote:

The perf has basic kernel ftrace support but lack support of most

tracing

Quoted 1 line

options. This serias is target to enhance the perf ftrace

functionality so

Quoted 4 lines

that we can make full use of kernel ftrace with only perf.

In general, this serias be cataloged into two main changes:
1) Improve usability of existing functions. For example, we don't

need to type

Quoted 19 lines

extra option to select the tracer.
2) Add new options to support all other ftrace functions.

Here is a glance of all ftrace functions with this serias:
* - improved existing options.
+ - new added options.

$ sudo perf ftrace -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- <command> [<options>]

* -a, --all-cpus system-wide collection from all CPUs

(quoted diff) +2 -0
+   -b, --buffer-size <n>
                          size of per cpu buffer in kb
    -C, --cpu <cpu>       list of cpus to monitor
+   -d, --delay <n>       Wait <n> ms before tracing
    -D, --graph-depth <n>
                          Max depth for function graph tracer
Quoted 2 lines

* -G, --graph-funcs <func>
Set graph filter on given functions (imply

to use function_graph tracer)

Quoted 2 lines

-g, --nograph-funcs <func>
Set nograph filter on given functions

(imply to use function_graph tracer)

Review

Quoted 2 lines
(quoted diff) +2 -0
+   -L, --list-functions  List available functions to filter
+   -l, --long-info       Show process names, PIDs, timestamps,

irq-info if available

Review

Quoted 7 lines

-N, --notrace-funcs <func>
do not trace given functions

(quoted diff) +3 -0
+   -P, --no-pager        Do not use pager
    -p, --pid <pid>       trace on existing process id
+   -s, --func-stack-trace
                          Show kernel stack trace for function tracer
+   -t, --tid <tid>       trace on existing thread id (exclusive to

--pid)

Quoted 3 lines

-T, --trace-funcs <func>
trace given functions only
+ -u, --userstacktrace Show stacktrace of the current user space

thread

Quoted 2 lines
    -v, --verbose         be more verbose

+ --funcgraph-tail Show function tails comment (function_graph

only)

Quoted 1 line

+ --latency-format displays additional information about the

latency (function_graph only)

Quoted 2 lines

+ --nofuncgraph-irqs
Ignore functions that happen inside

interrupt (function_graph only)

Quoted 1 line

+ --nosleep-time Measure on-CPU time only (function_graph

only)

Review

Quoted 3 lines
(quoted diff) +2 -0
+       --trace-children  Trace children processes
+       --tracing-thresh <n>
                          Only show functions of which the duration

is greater than <n>µs

Quoted 1 line

Note, we are working on making more of the trace-cmd functionality into libraries. See this work here:

https://lore.kernel.org/r/[email protected]

Quoted 8 lines

Which introduces a libtracefs, that is to handle all the work needed to interact with the tracefs directory. This will also be useful for perf to read the event directory without having to open code that work.

I'm all for giving perf the functionality of ftrace, but I would like to have it do so with a more generic solution that other tools could benefit from as well.

I think in time things will fall into place, 'perf ftrace' has potential as a way for perf users to have a familiar interface to ftrace, and that I think it's its raison d'être, in time well go on finding ways to have common code.

Tomorrow I'll go over this patch set,

Thanks,

- Arnaldo

Quoted 3 lines

Thanks!

-- Steve

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#56 lore raw
In reply to Changbin Du

Em Sun, May 10, 2020 at 11:06:09PM +0800, Changbin Du escreveu:

Quoted 3 lines

The perf has basic kernel ftrace support but lack support of most tracing options. This serias is target to enhance the perf ftrace functionality so that we can make full use of kernel ftrace with only perf.

Quite a lot! Going thru it now.

- Arnaldo

Quoted 79 lines

In general, this serias be cataloged into two main changes:
1) Improve usability of existing functions. For example, we don't need to type extra option to select the tracer.
2) Add new options to support all other ftrace functions.

Here is a glance of all ftrace functions with this serias:
* - improved existing options.
+ - new added options.

$ sudo perf ftrace -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- <command> [<options>]

* -a, --all-cpus system-wide collection from all CPUs

(quoted diff) +2 -0
+   -b, --buffer-size <n>
                          size of per cpu buffer in kb
    -C, --cpu <cpu>       list of cpus to monitor
+   -d, --delay <n>       Wait <n> ms before tracing
    -D, --graph-depth <n>
                          Max depth for function graph tracer

* -G, --graph-funcs <func>
Set graph filter on given functions (imply to use function_graph tracer)
-g, --nograph-funcs <func>
Set nograph filter on given functions (imply to use function_graph tracer)

(quoted diff) +12 -0
+   -L, --list-functions  List available functions to filter
+   -l, --long-info       Show process names, PIDs, timestamps, irq-info if available
    -N, --notrace-funcs <func>
                          do not trace given functions
+   -P, --no-pager        Do not use pager
    -p, --pid <pid>       trace on existing process id
+   -s, --func-stack-trace
                          Show kernel stack trace for function tracer
+   -t, --tid <tid>       trace on existing thread id (exclusive to --pid)
    -T, --trace-funcs <func>
                          trace given functions only
+   -u, --userstacktrace  Show stacktrace of the current user space thread
    -v, --verbose         be more verbose
+       --funcgraph-tail  Show function tails comment (function_graph only)
+       --latency-format  displays additional information about the latency (function_graph only)
+       --nofuncgraph-irqs
                          Ignore functions that happen inside interrupt (function_graph only)
+       --nosleep-time    Measure on-CPU time only (function_graph only)
+       --trace-children  Trace children processes
+       --tracing-thresh <n>
                          Only show functions of which the duration is greater than <n>µs

Changbin Du (19):
perf ftrace: trace system wide if no target is given
perf ftrace: detect workload failure
perf ftrace: select function/function_graph tracer automatically perf ftrace: add support for tracing option 'func_stack_trace' perf ftrace: add option '-l/--list-functions' to list available
functions
perf ftrace: add support for trace option sleep-time
perf ftrace: add support for trace option funcgraph-irqs perf ftrace: add option -l/--long-info to show more info perf ftrace: add support for trace option tracing_thresh perf ftrace: add support for trace option funcgraph-tail perf ftrace: add option '-u/--userstacktrace' to show userspace
stacktrace
perf ftrace: add support for tracing children processes perf ftrace: add option '-b/--buffer-size' to set per-cpu buffer size perf ftrace: add option -P/--no-pager to disable pager perf ftrace: show trace column header
perf ftrace: add option -t/--tid to filter by thread id perf ftrace: add option -d/--delay to delay tracing
perf ftrace: add option --latency-format to display more info about
delay
perf ftrace: add change log

 tools/perf/Documentation/perf-config.txt |   5 -
 tools/perf/builtin-ftrace.c              | 388 ++++++++++++++++++++---
 2 files changed, 345 insertions(+), 48 deletions(-)
Signature · 1 line
2.25.1
Signature · 1 line
- Arnaldo
#57 lore raw
Quoted 9 lines · Arnaldo Carvalho de Melo

Em Sun, May 10, 2020 at 11:06:09PM +0800, Changbin Du escreveu:

Quoted 3 lines

The perf has basic kernel ftrace support but lack support of most tracing options. This serias is target to enhance the perf ftrace functionality so that we can make full use of kernel ftrace with only perf.

Quite a lot! Going thru it now.

- Arnaldo

Hi Arnaldo, thanks for your detailed review. Sorry for slow response since I am just getting busy with my own affairs. I'll do the changes acoording to your comments one by one as soon as possilbe.

[snip..]

Signature · 2 lines
Cheers,
Changbin Du