Skip to contentSkip to conversation

perf ftrace: Add 'profile' subcommand (v1)

PATCH In review Namhyung Kim 3 participants 10 messages Jul 29, 2024 – Jul 31, 2024 perf events

lore
Share & tools
lore

Namhyung Kim

author maintainer
#0 lore raw

cover perf ftrace: Add 'profile' subcommand (v1)

Hello,

This is an attempt to extend perf ftrace command to show a kernel function profile using the function_graph tracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.

  $ sudo perf ftrace profile -- sync | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
      7638.372   7638.372   7638.372          1   __do_sys_sync
      7638.059   7638.059   7638.059          1   ksys_sync
      5893.959   1964.653   3747.963          3   iterate_supers
      5214.181    579.353   1688.752          9   schedule
      3585.773     44.269   3537.329         81   sync_inodes_one_sb
      3566.179     44.027   3537.078         81   sync_inodes_sb
      1976.901    247.113   1968.070          8   filemap_fdatawait_keep_errors
      1974.367    246.796   1967.895          8   __filemap_fdatawait_range
      1935.407     37.219   1157.627         52   folio_wait_writeback

While the kernel also provides the similar functionality IIRC under CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.

Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.

  $ sudo perf ftrace profile -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]

-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions using function_graph tracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)

The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor out check_ftrace_capable() perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-command

 tools/perf/Documentation/perf-ftrace.txt |  48 ++-
 tools/perf/builtin-ftrace.c              | 439 +++++++++++++++++++++--
 tools/perf/util/ftrace.h                 |   3 +
 3 files changed, 463 insertions(+), 27 deletions(-)
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
author maintainer
#1 lore raw

PATCH 1/4 perf ftrace: Add 'tail' option to --graph-opts

The 'graph-tail' option is to print function name as a comment at the end. This is useful when a large function is mixed with other functions (possibly from different CPUs).

For example,

  $ sudo perf ftrace -- perf stat true
  ...
   1)               |    get_unused_fd_flags() {
   1)               |      alloc_fd() {
   1)   0.178 us    |        _raw_spin_lock();
   1)   0.187 us    |        expand_files();
   1)   0.169 us    |        _raw_spin_unlock();
   1)   1.211 us    |      }
   1)   1.503 us    |    }
  $ sudo perf ftrace --graph-opts tail -- perf stat true
  ...
   1)               |    get_unused_fd_flags() {
   1)               |      alloc_fd() {
   1)   0.099 us    |        _raw_spin_lock();
   1)   0.083 us    |        expand_files();
   1)   0.081 us    |        _raw_spin_unlock();
   1)   0.601 us    |      } /* alloc_fd */
   1)   0.751 us    |    } /* get_unused_fd_flags */
---
 tools/perf/Documentation/perf-ftrace.txt |  1 +
 tools/perf/builtin-ftrace.c              | 18 ++++++++++++++++++
 tools/perf/util/ftrace.h                 |  1 +
 3 files changed, 20 insertions(+)
tools/perf/Documentation/perf-ftrace.txt +1 -0
index d780b93fcf87..7ea1645a13cf 100644
@@ -125,6 +125,7 @@ OPTIONS for 'perf ftrace trace'
 	  - verbose      - Show process names, PIDs, timestamps, etc.
 	  - thresh=<n>   - Setup trace duration threshold in microseconds.
 	  - depth=<n>    - Set max depth for function graph tracer to follow.
+	  - tail         - Print function name at the end.
 
 
 OPTIONS for 'perf ftrace latency'
tools/perf/builtin-ftrace.c +18 -0
index eb30c8eca488..33c52ebda2fd 100644
@@ -228,6 +228,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)
 	write_tracing_option_file("funcgraph-irqs", "1");
 	write_tracing_option_file("funcgraph-proc", "0");
 	write_tracing_option_file("funcgraph-abstime", "0");
+	write_tracing_option_file("funcgraph-tail", "0");
 	write_tracing_option_file("latency-format", "0");
 	write_tracing_option_file("irq-info", "0");
 }
@@ -464,6 +465,17 @@ static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)
 	return 0;
 }
 
+static int set_tracing_funcgraph_tail(struct perf_ftrace *ftrace)
+{
+	if (!ftrace->graph_tail)
+		return 0;
+
+	if (write_tracing_option_file("funcgraph-tail", "1") < 0)
+		return -1;
+
+	return 0;
+}
+
 static int set_tracing_thresh(struct perf_ftrace *ftrace)
 {
 	int ret;
@@ -540,6 +552,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)
 		return -1;
 	}
 
+	if (set_tracing_funcgraph_tail(ftrace) < 0) {
+		pr_err("failed to set tracing option funcgraph-tail\n");
+		return -1;
+	}
+
 	return 0;
 }
 
@@ -1099,6 +1116,7 @@ static int parse_graph_tracer_opts(const struct option *opt,
 		{ .name = "verbose",		.value_ptr = &ftrace->graph_verbose },
 		{ .name = "thresh",		.value_ptr = &ftrace->graph_thresh },
 		{ .name = "depth",		.value_ptr = &ftrace->graph_depth },
+		{ .name = "tail",		.value_ptr = &ftrace->graph_tail },
 		{ .name = NULL, }
 	};
 
tools/perf/util/ftrace.h +1 -0
index 558efcb98d25..2515e841c64c 100644
@@ -25,6 +25,7 @@ struct perf_ftrace {
 	int			graph_noirqs;
 	int			graph_verbose;
 	int			graph_thresh;
+	int			graph_tail;
 };
 
 struct filter_entry {
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
author maintainer
#2 lore raw

PATCH 2/4 perf ftrace: Factor out check_ftrace_capable()

In reply to Namhyung Kim

The check is a common part of the ftrace commands, let's move it out.

---
 tools/perf/builtin-ftrace.c | 44 +++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 24 deletions(-)
tools/perf/builtin-ftrace.c +20 -24
index 33c52ebda2fd..978608ecd89c 100644
@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused,
 	done = true;
 }
 
+static int check_ftrace_capable(void)
+{
+	if (!(perf_cap__capable(CAP_PERFMON) ||
+	      perf_cap__capable(CAP_SYS_ADMIN))) {
+		pr_err("ftrace only works for %s!\n",
+#ifdef HAVE_LIBCAP_SUPPORT
+		"users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
+#else
+		"root"
+#endif
+		);
+		return -1;
+	}
+	return 0;
+}
+
 static int __write_tracing_file(const char *name, const char *val, bool append)
 {
 	char *file;
@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
 		.events = POLLIN,
 	};
 
-	if (!(perf_cap__capable(CAP_PERFMON) ||
-	      perf_cap__capable(CAP_SYS_ADMIN))) {
-		pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-		"users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-		"root"
-#endif
-		);
-		return -1;
-	}
-
 	select_tracer(ftrace);
 
 	if (reset_tracing_files(ftrace) < 0) {
@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace)
 	};
 	int buckets[NUM_BUCKET] = { };
 
-	if (!(perf_cap__capable(CAP_PERFMON) ||
-	      perf_cap__capable(CAP_SYS_ADMIN))) {
-		pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-		"users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-		"root"
-#endif
-		);
-		return -1;
-	}
-
 	trace_fd = prepare_func_latency(ftrace);
 	if (trace_fd < 0)
 		goto out;
@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)
 	signal(SIGCHLD, sig_handler);
 	signal(SIGPIPE, sig_handler);
 
+	ret = check_ftrace_capable();
+	if (ret < 0)
+		return -1;
+
 	ret = perf_config(perf_ftrace_config, &ftrace);
 	if (ret < 0)
 		return -1;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
reviewer
#3 lore raw

Re: [PATCH 2/4] perf ftrace: Factor out check_ftrace_capable()

Quoted 4 lines · Namhyung Kim

The check is a common part of the ftrace commands, let's move it out.

The whole ifdefs and perf_cap__capable seemed wrong, not least as we'd like the error message to be accurate not just when libcap is present, so I sent out;
https://lore.kernel.org/lkml/[email protected]/
which I think can supersede this.

Thanks,
Ian

Quoted 83 lines
---
 tools/perf/builtin-ftrace.c | 44 +++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 24 deletions(-)
tools/perf/builtin-ftrace.c +20 -25
index 33c52ebda2fd..978608ecd89c 100644
@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused,
        done = true;
 }
 
+static int check_ftrace_capable(void)
+{
+       if (!(perf_cap__capable(CAP_PERFMON) ||
+             perf_cap__capable(CAP_SYS_ADMIN))) {
+               pr_err("ftrace only works for %s!\n",
+#ifdef HAVE_LIBCAP_SUPPORT
+               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
+#else
+               "root"
+#endif
+               );
+               return -1;
+       }
+       return 0;
+}
+
 static int __write_tracing_file(const char *name, const char *val, bool append)
 {
        char *file;
@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
                .events = POLLIN,
        };
 
-       if (!(perf_cap__capable(CAP_PERFMON) ||
-             perf_cap__capable(CAP_SYS_ADMIN))) {
-               pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-               "root"
-#endif
-               );
-               return -1;
-       }
-
        select_tracer(ftrace);
 
        if (reset_tracing_files(ftrace) < 0) {
@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace)
        };
        int buckets[NUM_BUCKET] = { };
 
-       if (!(perf_cap__capable(CAP_PERFMON) ||
-             perf_cap__capable(CAP_SYS_ADMIN))) {
-               pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-               "root"
-#endif
-               );
-               return -1;
-       }
-
        trace_fd = prepare_func_latency(ftrace);
        if (trace_fd < 0)
                goto out;
@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)
        signal(SIGCHLD, sig_handler);
        signal(SIGPIPE, sig_handler);
 
+       ret = check_ftrace_capable();
+       if (ret < 0)
+               return -1;
+
        ret = perf_config(perf_ftrace_config, &ftrace);
        if (ret < 0)
                return -1;
--

2.46.0.rc1.232.g9752f9e123-goog

author maintainer
#4 lore raw

Re: [PATCH 2/4] perf ftrace: Factor out check_ftrace_capable()

Hi Ian,

Quoted 5 lines · Ian Rogers
Quoted 3 lines · Namhyung Kim

The check is a common part of the ftrace commands, let's move it out.

Quoted 7 lines
Quoted 1 line

The whole ifdefs and perf_cap__capable seemed wrong, not least as we'd like the error message to be accurate not just when libcap is present, so I sent out;
https://lore.kernel.org/lkml/[email protected]/
which I think can supersede this.

Thanks for the change! I'll reply to the patch.
Namhyung

Quoted 84 lines
Quoted 84 lines
---
 tools/perf/builtin-ftrace.c | 44 +++++++++++++++++--------------------
 1 file changed, 20 insertions(+), 24 deletions(-)
tools/perf/builtin-ftrace.c +20 -25
index 33c52ebda2fd..978608ecd89c 100644
@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused,
        done = true;
 }
 
+static int check_ftrace_capable(void)
+{
+       if (!(perf_cap__capable(CAP_PERFMON) ||
+             perf_cap__capable(CAP_SYS_ADMIN))) {
+               pr_err("ftrace only works for %s!\n",
+#ifdef HAVE_LIBCAP_SUPPORT
+               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
+#else
+               "root"
+#endif
+               );
+               return -1;
+       }
+       return 0;
+}
+
 static int __write_tracing_file(const char *name, const char *val, bool append)
 {
        char *file;
@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace)
                .events = POLLIN,
        };
 
-       if (!(perf_cap__capable(CAP_PERFMON) ||
-             perf_cap__capable(CAP_SYS_ADMIN))) {
-               pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-               "root"
-#endif
-               );
-               return -1;
-       }
-
        select_tracer(ftrace);
 
        if (reset_tracing_files(ftrace) < 0) {
@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace)
        };
        int buckets[NUM_BUCKET] = { };
 
-       if (!(perf_cap__capable(CAP_PERFMON) ||
-             perf_cap__capable(CAP_SYS_ADMIN))) {
-               pr_err("ftrace only works for %s!\n",
-#ifdef HAVE_LIBCAP_SUPPORT
-               "users with the CAP_PERFMON or CAP_SYS_ADMIN capability"
-#else
-               "root"
-#endif
-               );
-               return -1;
-       }
-
        trace_fd = prepare_func_latency(ftrace);
        if (trace_fd < 0)
                goto out;
@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)
        signal(SIGCHLD, sig_handler);
        signal(SIGPIPE, sig_handler);
 
+       ret = check_ftrace_capable();
+       if (ret < 0)
+               return -1;
+
        ret = perf_config(perf_ftrace_config, &ftrace);
        if (ret < 0)
                return -1;
--

2.46.0.rc1.232.g9752f9e123-goog

author maintainer
#5 lore raw

PATCH 3/4 perf ftrace: Add 'profile' command

In reply to Namhyung Kim

The 'perf ftrace profile' command is to get function execution profiles using function-graph tracer so that users can see the total, average, max execution time as well as the number of invocation easily.

The following is a profile for perf_event_open syscall.

  $ sudo perf ftrace profile -G __x64_sys_perf_event_open -- \
    perf stat -e cycles -C1 true 2> /dev/null | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
        65.611     65.611     65.611          1   __x64_sys_perf_event_open
        30.527     30.527     30.527          1   anon_inode_getfile
        30.260     30.260     30.260          1   __anon_inode_getfile
        29.700     29.700     29.700          1   alloc_file_pseudo
        17.578     17.578     17.578          1   d_alloc_pseudo
        17.382     17.382     17.382          1   __d_alloc
        16.738     16.738     16.738          1   kmem_cache_alloc_lru
        15.686     15.686     15.686          1   perf_event_alloc
        14.012      7.006     11.264          2   obj_cgroup_charge
---
 tools/perf/Documentation/perf-ftrace.txt |  42 ++-
 tools/perf/builtin-ftrace.c              | 318 ++++++++++++++++++++++-
 tools/perf/util/ftrace.h                 |   2 +
 3 files changed, 359 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +41 -1
index 7ea1645a13cf..33f32467f287 100644
@@ -9,7 +9,7 @@ perf-ftrace - simple wrapper for kernel's ftrace functionality
 SYNOPSIS
 --------
 [verse]
-'perf ftrace' {trace|latency} <command>
+'perf ftrace' {trace|latency|profile} <command>
 
 DESCRIPTION
 -----------
@@ -23,6 +23,9 @@ kernel's ftrace infrastructure.
   'perf ftrace latency' calculates execution latency of a given function
   (optionally with BPF) and display it as a histogram.
 
+  'perf ftrace profile' show a execution profile for each function including
+  total, average, max time and the number of calls.
+
 The following options apply to perf ftrace.
 
 COMMON OPTIONS
@@ -146,6 +149,43 @@ OPTIONS for 'perf ftrace latency'
 	Use nano-second instead of micro-second as a base unit of the histogram.
 
 
+OPTIONS for 'perf ftrace profile'
+---------------------------------
+
+-T::
+--trace-funcs=::
+	Set function filter on the given function (or a glob pattern).
+	Multiple functions can be given by using this option more than once.
+	The function argument also can be a glob pattern. It will be passed
+	to 'set_ftrace_filter' in tracefs.
+
+-N::
+--notrace-funcs=::
+	Do not trace functions given by the argument.  Like -T option, this
+	can be used more than once to specify multiple functions (or glob
+	patterns).  It will be passed to 'set_ftrace_notrace' in tracefs.
+
+-G::
+--graph-funcs=::
+	Set graph filter on the given function (or a glob pattern). This is
+	useful to trace for functions executed from the given function. This
+	can be used more than once to specify multiple functions. It will be
+	passed to 'set_graph_function' in tracefs.
+
+-g::
+--nograph-funcs=::
+	Set graph notrace filter on the given function (or a glob pattern).
+	Like -G option, this is useful for the function_graph tracer only and
+	disables tracing for function executed from the given function. This
+	can be used more than once to specify multiple functions. It will be
+	passed to 'set_graph_notrace' in tracefs.
+
+-m::
+--buffer-size::
+	Set the size of per-cpu tracing buffer, <size> is expected to
+	be a number with appended unit character - B/K/M/G.
+
+
 SEE ALSO
 --------
 linkperf:perf-record[1], linkperf:perf-trace[1]
tools/perf/builtin-ftrace.c +316 -2

374 lines not on this page · raw message

tools/perf/util/ftrace.h +2 -0
index 2515e841c64c..bae649ef50e8 100644
@@ -6,6 +6,7 @@
 #include "target.h"
 
 struct evlist;
+struct hashamp;
 
 struct perf_ftrace {
 	struct evlist		*evlist;
@@ -15,6 +16,7 @@ struct perf_ftrace {
 	struct list_head	notrace;
 	struct list_head	graph_funcs;
 	struct list_head	nograph_funcs;
+	struct hashmap		*profile_hash;
 	unsigned long		percpu_buffer_size;
 	bool			inherit;
 	bool			use_nsec;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
author maintainer
#6 lore raw

PATCH 4/4 perf ftrace: Add -s/--sort option to profile sub-command

In reply to Namhyung Kim

The -s/--sort option is to sort the output by given column.

  $ sudo perf ftrace profile -s max sync | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
      6301.811   6301.811   6301.811          1   __do_sys_sync
      6301.328   6301.328   6301.328          1   ksys_sync
      5320.300   1773.433   2858.819          3   iterate_supers
      2755.875     17.012   2610.633        162   sync_fs_one_sb
      2728.351    682.088   2610.413          4   ext4_sync_fs [ext4]
      2603.654   2603.654   2603.654          1   jbd2_log_wait_commit [jbd2]
      4750.615    593.827   2597.427          8   schedule
      2164.986     26.728   2115.673         81   sync_inodes_one_sb
      2143.842     26.467   2115.438         81   sync_inodes_sb
---
 tools/perf/Documentation/perf-ftrace.txt |  5 ++
 tools/perf/builtin-ftrace.c              | 63 +++++++++++++++++++++++-
 2 files changed, 66 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +5 -0
index 33f32467f287..eaec8253be68 100644
@@ -185,6 +185,11 @@ OPTIONS for 'perf ftrace profile'
 	Set the size of per-cpu tracing buffer, <size> is expected to
 	be a number with appended unit character - B/K/M/G.
 
+-s::
+--sort=::
+	Sort the result by the given field.  Available values are:
+	total, avg, max, count, name.  Default is 'total'.
+
 
 SEE ALSO
 --------
tools/perf/builtin-ftrace.c +61 -2

88 lines not on this page · raw message

Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
reviewer Reviewed-by on the v1 series
#7 lore raw
In reply to Namhyung Kim
Quoted 52 lines · Namhyung Kim

Hello,

This is an attempt to extend perf ftrace command to show a kernel function profile using the function_graph tracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.

  $ sudo perf ftrace profile -- sync | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
      7638.372   7638.372   7638.372          1   __do_sys_sync
      7638.059   7638.059   7638.059          1   ksys_sync
      5893.959   1964.653   3747.963          3   iterate_supers
      5214.181    579.353   1688.752          9   schedule
      3585.773     44.269   3537.329         81   sync_inodes_one_sb
      3566.179     44.027   3537.078         81   sync_inodes_sb
      1976.901    247.113   1968.070          8   filemap_fdatawait_keep_errors
      1974.367    246.796   1967.895          8   __filemap_fdatawait_range
      1935.407     37.219   1157.627         52   folio_wait_writeback

While the kernel also provides the similar functionality IIRC under CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.

Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.

  $ sudo perf ftrace profile -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]

-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions using function_graph tracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)

Quoted 5 lines

The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Lgtm, need to think about (rebase, etc) wrt the libcap change I sent but otherwise:

Thanks,
Ian

Quoted 14 lines

Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor out check_ftrace_capable() perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-command

 tools/perf/Documentation/perf-ftrace.txt |  48 ++-
 tools/perf/builtin-ftrace.c              | 439 +++++++++++++++++++++--
 tools/perf/util/ftrace.h                 |   3 +
 3 files changed, 463 insertions(+), 27 deletions(-)

--
2.46.0.rc1.232.g9752f9e123-goog

#8 lore raw
In reply to Namhyung Kim
Quoted 19 lines · Namhyung Kim

Hello,

This is an attempt to extend perf ftrace command to show a kernel function profile using the function_graph tracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.

  $ sudo perf ftrace profile -- sync | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
      7638.372   7638.372   7638.372          1   __do_sys_sync
      7638.059   7638.059   7638.059          1   ksys_sync
      5893.959   1964.653   3747.963          3   iterate_supers
      5214.181    579.353   1688.752          9   schedule
      3585.773     44.269   3537.329         81   sync_inodes_one_sb
      3566.179     44.027   3537.078         81   sync_inodes_sb
      1976.901    247.113   1968.070          8   filemap_fdatawait_keep_errors
      1974.367    246.796   1967.895          8   __filemap_fdatawait_range
      1935.407     37.219   1157.627         52   folio_wait_writeback
Quoted 3 lines

While the kernel also provides the similar functionality IIRC under CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.

Great functionality, tested it all and applied to tmp.perf-tools-next, will be in perf-tools-next after one last round of container builds.

The discussion about libcap seems to still be open, so I'm applying what is in this series as it is small and simple, we can go on from there.

Thanks!

- Arnaldo

Quoted 48 lines

Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.

  $ sudo perf ftrace profile -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]

-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions using function_graph tracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)

The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor out check_ftrace_capable() perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-command

 tools/perf/Documentation/perf-ftrace.txt |  48 ++-
 tools/perf/builtin-ftrace.c              | 439 +++++++++++++++++++++--
 tools/perf/util/ftrace.h                 |   3 +
 3 files changed, 463 insertions(+), 27 deletions(-)
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
reviewer
#9 lore raw
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 19 lines · Namhyung Kim

Hello,

This is an attempt to extend perf ftrace command to show a kernel function profile using the function_graph tracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.

  $ sudo perf ftrace profile -- sync | head
  # Total (us)   Avg (us)   Max (us)      Count   Function
      7638.372   7638.372   7638.372          1   __do_sys_sync
      7638.059   7638.059   7638.059          1   ksys_sync
      5893.959   1964.653   3747.963          3   iterate_supers
      5214.181    579.353   1688.752          9   schedule
      3585.773     44.269   3537.329         81   sync_inodes_one_sb
      3566.179     44.027   3537.078         81   sync_inodes_sb
      1976.901    247.113   1968.070          8   filemap_fdatawait_keep_errors
      1974.367    246.796   1967.895          8   __filemap_fdatawait_range
      1935.407     37.219   1157.627         52   folio_wait_writeback
Quoted 3 lines

While the kernel also provides the similar functionality IIRC under CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.

Quoted 5 lines

Great functionality, tested it all and applied to tmp.perf-tools-next, will be in perf-tools-next after one last round of container builds.

The discussion about libcap seems to still be open, so I'm applying what is in this series as it is small and simple, we can go on from there.

Sgtm. I did the libcap cleanup on perf-tools-next, but tmp.perf-tools-next hasn't been merged there for 3 weeks. It'd be nice to rebase the patch on perf-tools-next, but should I just shift to working on tmp.perf-tools-next?

Thanks,
Ian

Quoted 52 lines

Thanks!

- Arnaldo

Quoted 48 lines

Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.

  $ sudo perf ftrace profile -h

Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]

-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions using function_graph tracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)

The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git

Thanks,
Namhyung

Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor out check_ftrace_capable() perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-command

 tools/perf/Documentation/perf-ftrace.txt |  48 ++-
 tools/perf/builtin-ftrace.c              | 439 +++++++++++++++++++++--
 tools/perf/util/ftrace.h                 |   3 +
 3 files changed, 463 insertions(+), 27 deletions(-)

--
2.46.0.rc1.232.g9752f9e123-goog