perf ftrace: Add 'profile' subcommand (v1)
Namhyung Kim
maintainercover perf ftrace: Add 'profile' subcommand (v1)
Hello,
This is an attempt to extend perf ftrace command to show a kernel function profile using the function_graph tracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.
$ sudo perf ftrace profile -- sync | head
# Total (us) Avg (us) Max (us) Count Function
7638.372 7638.372 7638.372 1 __do_sys_sync
7638.059 7638.059 7638.059 1 ksys_sync
5893.959 1964.653 3747.963 3 iterate_supers
5214.181 579.353 1688.752 9 schedule
3585.773 44.269 3537.329 81 sync_inodes_one_sb
3566.179 44.027 3537.078 81 sync_inodes_sb
1976.901 247.113 1968.070 8 filemap_fdatawait_keep_errors
1974.367 246.796 1967.895 8 __filemap_fdatawait_range
1935.407 37.219 1157.627 52 folio_wait_writebackWhile the kernel also provides the similar functionality IIRC under CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.
Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.
$ sudo perf ftrace profile -h
Usage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]
-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions using function_graph tracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)
The code is also available in 'perf/ftrace-profile-v1' branch atgit://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor out check_ftrace_capable() perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-command
tools/perf/Documentation/perf-ftrace.txt | 48 ++- tools/perf/builtin-ftrace.c | 439 +++++++++++++++++++++-- tools/perf/util/ftrace.h | 3 + 3 files changed, 463 insertions(+), 27 deletions(-)
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Namhyung Kim
maintainerPATCH 1/4 perf ftrace: Add 'tail' option to --graph-opts
The 'graph-tail' option is to print function name as a comment at the end. This is useful when a large function is mixed with other functions (possibly from different CPUs).
For example,
$ sudo perf ftrace -- perf stat true
...
1) | get_unused_fd_flags() {
1) | alloc_fd() {
1) 0.178 us | _raw_spin_lock();
1) 0.187 us | expand_files();
1) 0.169 us | _raw_spin_unlock();
1) 1.211 us | }
1) 1.503 us | } $ sudo perf ftrace --graph-opts tail -- perf stat true
...
1) | get_unused_fd_flags() {
1) | alloc_fd() {
1) 0.099 us | _raw_spin_lock();
1) 0.083 us | expand_files();
1) 0.081 us | _raw_spin_unlock();
1) 0.601 us | } /* alloc_fd */
1) 0.751 us | } /* get_unused_fd_flags */--- tools/perf/Documentation/perf-ftrace.txt | 1 + tools/perf/builtin-ftrace.c | 18 ++++++++++++++++++ tools/perf/util/ftrace.h | 1 + 3 files changed, 20 insertions(+)
tools/perf/Documentation/perf-ftrace.txt +1 -0
@@ -125,6 +125,7 @@ OPTIONS for 'perf ftrace trace' - verbose - Show process names, PIDs, timestamps, etc. - thresh=<n> - Setup trace duration threshold in microseconds. - depth=<n> - Set max depth for function graph tracer to follow. + - tail - Print function name at the end. OPTIONS for 'perf ftrace latency'
tools/perf/builtin-ftrace.c +18 -0
@@ -228,6 +228,7 @@ static void reset_tracing_options(struct perf_ftrace *ftrace __maybe_unused)write_tracing_option_file ("funcgraph-irqs" ,"1" );write_tracing_option_file ("funcgraph-proc" ,"0" );write_tracing_option_file ("funcgraph-abstime" ,"0" ); +write_tracing_option_file ("funcgraph-tail" ,"0" );write_tracing_option_file ("latency-format" ,"0" );write_tracing_option_file ("irq-info" ,"0" ); }
@@ -464,6 +465,17 @@ static int set_tracing_funcgraph_verbose(struct perf_ftrace *ftrace)return 0 ; } +static int set_tracing_funcgraph_tail (struct perf_ftrace *ftrace) +{ +if (!ftrace->graph_tail) +return 0 ; + +if (write_tracing_option_file ("funcgraph-tail" ,"1" ) <0 ) +return -1 ; + +return 0 ; +} +static int set_tracing_thresh (struct perf_ftrace *ftrace) {int ret;
@@ -540,6 +552,11 @@ static int set_tracing_options(struct perf_ftrace *ftrace)return -1 ; } +if (set_tracing_funcgraph_tail (ftrace) <0 ) { +pr_err ("failed to set tracing option funcgraph-tail\n" ); +return -1 ; + } +return 0 ; }
@@ -1099,6 +1116,7 @@ static int parse_graph_tracer_opts(const struct option *opt, { .name = "verbose", .value_ptr = &ftrace->graph_verbose }, { .name = "thresh", .value_ptr = &ftrace->graph_thresh }, { .name = "depth", .value_ptr = &ftrace->graph_depth }, + { .name = "tail", .value_ptr = &ftrace->graph_tail }, { .name = NULL, } };
tools/perf/util/ftrace.h +1 -0
@@ -25,6 +25,7 @@ struct perf_ftrace {int graph_noirqs;int graph_verbose;int graph_thresh; +int graph_tail; };struct filter_entry {
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Namhyung Kim
maintainerPATCH 2/4 perf ftrace: Factor out check_ftrace_capable()
In reply to Namhyung KimThe check is a common part of the ftrace commands, let's move it out.
--- tools/perf/builtin-ftrace.c | 44 +++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 24 deletions(-)
tools/perf/builtin-ftrace.c +20 -24
@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused, done = true; } +static int check_ftrace_capable (void ) +{ +if (!(perf_cap__capable (CAP_PERFMON ) || +perf_cap__capable (CAP_SYS_ADMIN ))) { +pr_err ("ftrace only works for %s!\n" , +#ifdef HAVE_LIBCAP_SUPPORT + "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" +#else +"root" +#endif + ); +return -1 ; + } +return 0 ; +} +static int __write_tracing_file (const char *name,const char *val,bool append) {char *file;
@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) .events = POLLIN, }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } -select_tracer (ftrace);if (reset_tracing_files (ftrace) <0 ) {
@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace) };int buckets[NUM_BUCKET ] = { }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } - trace_fd =prepare_func_latency (ftrace);if (trace_fd <0 ) goto out;
@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)signal (SIGCHLD , sig_handler);signal (SIGPIPE , sig_handler); + ret =check_ftrace_capable (); +if (ret <0 ) +return -1 ; + ret =perf_config (perf_ftrace_config, &ftrace);if (ret <0 )return -1 ;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Ian Rogers
reviewerRe: [PATCH 2/4] perf ftrace: Factor out check_ftrace_capable()
Quoted 4 lines · Namhyung Kim
The check is a common part of the ftrace commands, let's move it out.
Signed-off-by Namhyung Kim <[email protected]>
The whole ifdefs and perf_cap__capable seemed wrong, not least as we'd like the error message to be accurate not just when libcap is present, so I sent out;
https://lore.kernel.org/lkml/[email protected]/
which I think can supersede this.
Thanks,
Ian
Quoted 83 lines
--- tools/perf/builtin-ftrace.c | 44 +++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 24 deletions(-)tools/perf/builtin-ftrace.c +20
-25@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused, done = true; } +static int check_ftrace_capable (void ) +{ +if (!(perf_cap__capable (CAP_PERFMON ) || +perf_cap__capable (CAP_SYS_ADMIN ))) { +pr_err ("ftrace only works for %s!\n" , +#ifdef HAVE_LIBCAP_SUPPORT + "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" +#else +"root" +#endif + ); +return -1 ; + } +return 0 ; +} +static int __write_tracing_file (const char *name,const char *val,bool append) {char *file;@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) .events = POLLIN, }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } -select_tracer (ftrace);if (reset_tracing_files (ftrace) <0 ) {@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace) };int buckets[NUM_BUCKET ] = { }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } - trace_fd =prepare_func_latency (ftrace);if (trace_fd <0 ) goto out;@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)signal (SIGCHLD , sig_handler);signal (SIGPIPE , sig_handler); + ret =check_ftrace_capable (); +if (ret <0 ) +return -1 ; + ret =perf_config (perf_ftrace_config, &ftrace);if (ret <0 )return -1 ; --2.46.0.rc1.232.g9752f9e123-goog
Namhyung Kim
maintainerRe: [PATCH 2/4] perf ftrace: Factor out check_ftrace_capable()
Hi Ian,
Quoted 5 lines · Ian Rogers
Quoted 3 lines · Namhyung Kim
The check is a common part of the ftrace commands, let's move it out.
Quoted 7 lines
Quoted 1 line
Signed-off-by Namhyung Kim <[email protected]>The whole ifdefs and
perf_cap__capableseemed wrong, not least as we'd like the error message to be accurate not just when libcap is present, so I sent out;
https://lore.kernel.org/lkml/[email protected]/
which I think can supersede this.
Thanks for the change! I'll reply to the patch.
Namhyung
Quoted 84 lines
Quoted 84 lines
--- tools/perf/builtin-ftrace.c | 44 +++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 24 deletions(-)tools/perf/builtin-ftrace.c +20
-25@@ -59,6 +59,22 @@ static void ftrace__workload_exec_failed_signal(int signo __maybe_unused, done = true; } +static int check_ftrace_capable (void ) +{ +if (!(perf_cap__capable (CAP_PERFMON ) || +perf_cap__capable (CAP_SYS_ADMIN ))) { +pr_err ("ftrace only works for %s!\n" , +#ifdef HAVE_LIBCAP_SUPPORT + "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" +#else +"root" +#endif + ); +return -1 ; + } +return 0 ; +} +static int __write_tracing_file (const char *name,const char *val,bool append) {char *file;@@ -586,18 +602,6 @@ static int __cmd_ftrace(struct perf_ftrace *ftrace) .events = POLLIN, }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } -select_tracer (ftrace);if (reset_tracing_files (ftrace) <0 ) {@@ -902,18 +906,6 @@ static int __cmd_latency(struct perf_ftrace *ftrace) };int buckets[NUM_BUCKET ] = { }; -if (!(perf_cap__capable (CAP_PERFMON ) || -perf_cap__capable (CAP_SYS_ADMIN ))) { -pr_err ("ftrace only works for %s!\n" , -#ifdef HAVE_LIBCAP_SUPPORT - "users with the CAP_PERFMON or CAP_SYS_ADMIN capability" -#else -"root" -#endif - ); -return -1 ; - } - trace_fd =prepare_func_latency (ftrace);if (trace_fd <0 ) goto out;@@ -1220,6 +1212,10 @@ int cmd_ftrace(int argc, const char **argv)signal (SIGCHLD , sig_handler);signal (SIGPIPE , sig_handler); + ret =check_ftrace_capable (); +if (ret <0 ) +return -1 ; + ret =perf_config (perf_ftrace_config, &ftrace);if (ret <0 )return -1 ; --2.46.0.rc1.232.g9752f9e123-goog
Namhyung Kim
maintainerPATCH 3/4 perf ftrace: Add 'profile' command
In reply to Namhyung KimThe 'perf ftrace profile' command is to get function execution profiles using function-graph tracer so that users can see the total, average, max execution time as well as the number of invocation easily.
The following is a profile for perf_event_open syscall.
$ sudo perf ftrace profile -G __x64_sys_perf_event_open -- \
perf stat -e cycles -C1 true 2> /dev/null | head
# Total (us) Avg (us) Max (us) Count Function
65.611 65.611 65.611 1 __x64_sys_perf_event_open
30.527 30.527 30.527 1 anon_inode_getfile
30.260 30.260 30.260 1 __anon_inode_getfile
29.700 29.700 29.700 1 alloc_file_pseudo
17.578 17.578 17.578 1 d_alloc_pseudo
17.382 17.382 17.382 1 __d_alloc
16.738 16.738 16.738 1 kmem_cache_alloc_lru
15.686 15.686 15.686 1 perf_event_alloc
14.012 7.006 11.264 2 obj_cgroup_charge--- tools/perf/Documentation/perf-ftrace.txt | 42 ++- tools/perf/builtin-ftrace.c | 318 ++++++++++++++++++++++- tools/perf/util/ftrace.h | 2 + 3 files changed, 359 insertions(+), 3 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +41 -1
@@ -9,7 +9,7 @@ perf-ftrace - simple wrapper for kernel's ftrace functionality SYNOPSIS -------- [verse] -'perf ftrace' {trace|latency} <command> +'perf ftrace' {trace|latency|profile} <command> DESCRIPTION -----------
@@ -23,6 +23,9 @@ kernel's ftrace infrastructure. 'perf ftrace latency' calculates execution latency of a given function (optionally with BPF) and display it as a histogram. + 'perf ftrace profile' show a execution profile for each function including + total, average, max time and the number of calls. + The following options apply to perf ftrace. COMMON OPTIONS
@@ -146,6 +149,43 @@ OPTIONS for 'perf ftrace latency' Use nano-second instead of micro-second as a base unit of the histogram. +OPTIONS for 'perf ftrace profile' +--------------------------------- + +-T:: +--trace-funcs=:: + Set function filter on the given function (or a glob pattern). + Multiple functions can be given by using this option more than once. + The function argument also can be a glob pattern. It will be passed + to 'set_ftrace_filter' in tracefs. + +-N:: +--notrace-funcs=:: + Do not trace functions given by the argument. Like -T option, this + can be used more than once to specify multiple functions (or glob + patterns). It will be passed to 'set_ftrace_notrace' in tracefs. + +-G:: +--graph-funcs=:: + Set graph filter on the given function (or a glob pattern). This is + useful to trace for functions executed from the given function. This + can be used more than once to specify multiple functions. It will be + passed to 'set_graph_function' in tracefs. + +-g:: +--nograph-funcs=:: + Set graph notrace filter on the given function (or a glob pattern). + Like -G option, this is useful for the function_graph tracer only and + disables tracing for function executed from the given function. This + can be used more than once to specify multiple functions. It will be + passed to 'set_graph_notrace' in tracefs. + +-m:: +--buffer-size:: + Set the size of per-cpu tracing buffer, <size> is expected to + be a number with appended unit character - B/K/M/G. + + SEE ALSO -------- linkperf:perf-record[1], linkperf:perf-trace[1]
tools/perf/builtin-ftrace.c +316 -2
374 lines not on this page · raw message
tools/perf/util/ftrace.h +2 -0
@@ -6,6 +6,7 @@#include "target.h" struct evlist ; +struct hashamp ;struct perf_ftrace {struct evlist *evlist;
@@ -15,6 +16,7 @@ struct perf_ftrace {struct list_head notrace;struct list_head graph_funcs;struct list_head nograph_funcs; +struct hashmap *profile_hash;unsigned long percpu_buffer_size;bool inherit;bool use_nsec;
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Namhyung Kim
maintainerPATCH 4/4 perf ftrace: Add -s/--sort option to profile sub-command
In reply to Namhyung KimThe -s/--sort option is to sort the output by given column.
$ sudo perf ftrace profile -s max sync | head
# Total (us) Avg (us) Max (us) Count Function
6301.811 6301.811 6301.811 1 __do_sys_sync
6301.328 6301.328 6301.328 1 ksys_sync
5320.300 1773.433 2858.819 3 iterate_supers
2755.875 17.012 2610.633 162 sync_fs_one_sb
2728.351 682.088 2610.413 4 ext4_sync_fs [ext4]
2603.654 2603.654 2603.654 1 jbd2_log_wait_commit [jbd2]
4750.615 593.827 2597.427 8 schedule
2164.986 26.728 2115.673 81 sync_inodes_one_sb
2143.842 26.467 2115.438 81 sync_inodes_sb--- tools/perf/Documentation/perf-ftrace.txt | 5 ++ tools/perf/builtin-ftrace.c | 63 +++++++++++++++++++++++- 2 files changed, 66 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-ftrace.txt +5 -0
@@ -185,6 +185,11 @@ OPTIONS for 'perf ftrace profile' Set the size of per-cpu tracing buffer, <size> is expected to be a number with appended unit character - B/K/M/G. +-s:: +--sort=:: + Sort the result by the given field. Available values are: + total, avg, max, count, name. Default is 'total'. + SEE ALSO --------
tools/perf/builtin-ftrace.c +61 -2
88 lines not on this page · raw message
Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Quoted 52 lines · Namhyung Kim
Hello,
This is an attempt to extend perf ftrace command to show a kernel function profile using the
function_graphtracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.$ sudo perf ftrace profile -- sync | head # Total (us) Avg (us) Max (us) Count Function 7638.372 7638.372 7638.372 1 __do_sys_sync 7638.059 7638.059 7638.059 1 ksys_sync 5893.959 1964.653 3747.963 3 iterate_supers 5214.181 579.353 1688.752 9 schedule 3585.773 44.269 3537.329 81 sync_inodes_one_sb 3566.179 44.027 3537.078 81 sync_inodes_sb 1976.901 247.113 1968.070 8 filemap_fdatawait_keep_errors 1974.367 246.796 1967.895 8 __filemap_fdatawait_range 1935.407 37.219 1157.627 52 folio_wait_writebackWhile the kernel also provides the similar functionality IIRC under
CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.
$ sudo perf ftrace profile -hUsage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions usingfunction_graphtracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)
Quoted 5 lines
The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
Namhyung
Lgtm, need to think about (rebase, etc) wrt the libcap change I sent but otherwise:
Thanks,
Ian
Quoted 14 lines
Namhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor outcheck_ftrace_capable()perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-commandtools/perf/Documentation/perf-ftrace.txt | 48 ++- tools/perf/builtin-ftrace.c | 439 +++++++++++++++++++++-- tools/perf/util/ftrace.h | 3 + 3 files changed, 463 insertions(+), 27 deletions(-)--
2.46.0.rc1.232.g9752f9e123-goog
Arnaldo Carvalho de Melo
maintainerQuoted 19 lines · Namhyung Kim
Hello,
This is an attempt to extend perf ftrace command to show a kernel function profile using the
function_graphtracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.$ sudo perf ftrace profile -- sync | head # Total (us) Avg (us) Max (us) Count Function 7638.372 7638.372 7638.372 1 __do_sys_sync 7638.059 7638.059 7638.059 1 ksys_sync 5893.959 1964.653 3747.963 3 iterate_supers 5214.181 579.353 1688.752 9 schedule 3585.773 44.269 3537.329 81 sync_inodes_one_sb 3566.179 44.027 3537.078 81 sync_inodes_sb 1976.901 247.113 1968.070 8 filemap_fdatawait_keep_errors 1974.367 246.796 1967.895 8 __filemap_fdatawait_range 1935.407 37.219 1157.627 52 folio_wait_writeback
Quoted 3 lines
While the kernel also provides the similar functionality IIRC under
CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.
Great functionality, tested it all and applied to tmp.perf-tools-next, will be in perf-tools-next after one last round of container builds.
The discussion about libcap seems to still be open, so I'm applying what is in this series as it is small and simple, we can go on from there.
Thanks!
- Arnaldo
Quoted 48 lines
Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.
$ sudo perf ftrace profile -hUsage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions usingfunction_graphtracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
NamhyungNamhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor outcheck_ftrace_capable()perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-commandtools/perf/Documentation/perf-ftrace.txt | 48 ++- tools/perf/builtin-ftrace.c | 439 +++++++++++++++++++++-- tools/perf/util/ftrace.h | 3 + 3 files changed, 463 insertions(+), 27 deletions(-)Signature · 1 line
2.46.0.rc1.232.g9752f9e123-goog
Ian Rogers
reviewerQuoted 25 lines · Arnaldo Carvalho de Melo
Quoted 19 lines · Namhyung Kim
Hello,
This is an attempt to extend perf ftrace command to show a kernel function profile using the
function_graphtracer. This is useful to see detailed info like total, average, max time (in usec) and number of calls for each function.$ sudo perf ftrace profile -- sync | head # Total (us) Avg (us) Max (us) Count Function 7638.372 7638.372 7638.372 1 __do_sys_sync 7638.059 7638.059 7638.059 1 ksys_sync 5893.959 1964.653 3747.963 3 iterate_supers 5214.181 579.353 1688.752 9 schedule 3585.773 44.269 3537.329 81 sync_inodes_one_sb 3566.179 44.027 3537.078 81 sync_inodes_sb 1976.901 247.113 1968.070 8 filemap_fdatawait_keep_errors 1974.367 246.796 1967.895 8 __filemap_fdatawait_range 1935.407 37.219 1157.627 52 folio_wait_writebackQuoted 3 lines
While the kernel also provides the similar functionality IIRC under
CONFIG_FUNCTION_PROFILER, it's often not enabled on disto kernels so I implemented it in user space.
Quoted 5 lines
Great functionality, tested it all and applied to tmp.perf-tools-next, will be in perf-tools-next after one last round of container builds.
The discussion about libcap seems to still be open, so I'm applying what is in this series as it is small and simple, we can go on from there.
Sgtm. I did the libcap cleanup on perf-tools-next, but tmp.perf-tools-next hasn't been merged there for 3 weeks. It'd be nice to rebase the patch on perf-tools-next, but should I just shift to working on tmp.perf-tools-next?
Thanks,
Ian
Quoted 52 lines
Thanks!
- Arnaldo
Quoted 48 lines
Also it can support function filters like 'perf ftrace trace' so users can focus on some target functions and change the buffer size if needed.
$ sudo perf ftrace profile -hUsage: perf ftrace [<options>] [<command>]
or: perf ftrace [<options>] -- [<command>] [<options>]
or: perf ftrace {trace|latency|profile} [<options>] [<command>]
or: perf ftrace {trace|latency|profile} [<options>] -- [<command>] [<options>]-a, --all-cpus System-wide collection from all CPUs
-C, --cpu <cpu> List of cpus to monitor
-G, --graph-funcs <func>
Trace given functions usingfunction_graphtracer
-g, --nograph-funcs <func>
Set nograph filter on given functions
-m, --buffer-size <size>
Size of per cpu buffer, needs to use a B, K, M or G suffix.
-N, --notrace-funcs <func>
Do not trace given functions
-p, --pid <pid> Trace on existing process id
-s, --sort <key> Sort result by key: total (default), avg, max, count, name.
-T, --trace-funcs <func>
Trace given functions using function tracer
-v, --verbose Be more verbose
--tid <tid> Trace on existing thread id (exclusive to --pid)The code is also available in 'perf/ftrace-profile-v1' branch at
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.gitThanks,
NamhyungNamhyung Kim (4):
perf ftrace: Add 'tail' option to --graph-opts perf ftrace: Factor outcheck_ftrace_capable()perf ftrace: Add 'profile' command
perf ftrace: Add -s/--sort option to profile sub-commandtools/perf/Documentation/perf-ftrace.txt | 48 ++- tools/perf/builtin-ftrace.c | 439 +++++++++++++++++++++-- tools/perf/util/ftrace.h | 3 + 3 files changed, 463 insertions(+), 27 deletions(-)--
2.46.0.rc1.232.g9752f9e123-goog