Make perf script easier to use for itrace
Implement a range of improvements to make it easier to look at itrace traces with perf script. Nothing here couldn't be done before with some additional scripting, but add simple high level options to make it easier to use.
% perf record -e intel_pt//k -a sleep 1
Show function calls:
% perf script --call-trace
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpageShow function calls and returns:
% perf script --call-ret-trace
perf 900 [000] 194167.205652203: tr strt ([unknown]) pt_config
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) pt_config
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) pt_event_add
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) perf_pmu_nop_void
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) perf_pmu_nop_int
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) group_sched_inShow instruction traces (using XED):
% perf script --insn-trace
swapper 0 [000] 117276.429606186: ffffffff81010486 pt_config ([kernel.kallsyms]) nopl %eax, (%rax,%rax,1)
swapper 0 [000] 117276.429606186: ffffffff8101048b pt_config ([kernel.kallsyms]) add $0x10, %rsp
swapper 0 [000] 117276.429606186: ffffffff8101048f pt_config ([kernel.kallsyms]) popq %rbx
swapper 0 [000] 117276.429606186: ffffffff81010490 pt_config ([kernel.kallsyms]) popq %rbp
swapper 0 [000] 117276.429606186: ffffffff81010491 pt_config ([kernel.kallsyms]) popq %r12
swapper 0 [000] 117276.429606186: ffffffff81010493 pt_config ([kernel.kallsyms]) popq %r13
swapper 0 [000] 117276.429606186: ffffffff81010495 pt_config ([kernel.kallsyms]) popq %r14
swapper 0 [000] 117276.429606186: ffffffff81010497 pt_config ([kernel.kallsyms]) popq %r15
swapper 0 [000] 117276.429606186: ffffffff81010499 pt_config ([kernel.kallsyms]) retqFilter by a ftrace style graph function:
% perf script --graph-function group_sched_in --call-trace
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpage
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) calc_timer_values
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_cpu
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) arch_perf_update_userpage
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __fentry__
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) using_native_sched_clock
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_stable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_raxAlso available ingit://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc.git pt/easy-script-2
v1: Initial post
v2: Address review comments. Minor fixes to descriptions. Now builds everywhere.
PATCH v2 1/10 perf tools: Report itrace options in help
From: Andi Kleen <[email protected]>
I often forget all the options that --itrace accepts. Instead of burying them in the man page only report them in the normal command line help too to make them easier accessible.
v2: Align
--- tools/perf/builtin-inject.c | 3 ++- tools/perf/builtin-report.c | 2 +- tools/perf/builtin-script.c | 2 +- tools/perf/util/auxtrace.h | 19 +++++++++++++++++++ 4 files changed, 23 insertions(+), 3 deletions(-)
tools/perf/builtin-inject.c +2 -1
@@ -803,7 +803,8 @@ int cmd_inject(int argc, const char **argv) "kallsyms pathname"),OPT_BOOLEAN ('f' ,"force" , &data.force,"don't complain, do it" ),OPT_CALLBACK_OPTARG (0 ,"itrace" , &inject.itrace_synth_opts, -NULL ,"opts" ,"Instruction Tracing options" , +NULL ,"opts" ,"Instruction Tracing options\n" +ITRACE_HELP , itrace_parse_synth_opts),OPT_BOOLEAN (0 ,"strip" , &inject.strip, "strip non-synthesized events (use with --itrace)"),
tools/perf/builtin-report.c +1 -1
@@ -1105,7 +1105,7 @@ int cmd_report(int argc, const char **argv)OPT_CALLBACK (0 ,"percentage" ,NULL ,"relative|absolute" ,"how to display percentage of filtered entries" , parse_filter_percentage),OPT_CALLBACK_OPTARG (0 ,"itrace" , &itrace_synth_opts,NULL ,"opts" , -"Instruction Tracing options" , +"Instruction Tracing options\n" ITRACE_HELP , itrace_parse_synth_opts),OPT_BOOLEAN (0 ,"full-source-path" , &srcline_full_filename, "Show full source file name path for source lines"),
tools/perf/builtin-script.c +1 -1
@@ -3193,7 +3193,7 @@ int cmd_script(int argc, const char **argv)OPT_BOOLEAN (0 ,"ns" , &nanosecs,"Use 9 decimal places when displaying time" ),OPT_CALLBACK_OPTARG (0 ,"itrace" , &itrace_synth_opts,NULL ,"opts" , -"Instruction Tracing options" , +"Instruction Tracing options\n" ITRACE_HELP , itrace_parse_synth_opts),OPT_BOOLEAN (0 ,"full-source-path" , &srcline_full_filename, "Show full source file name path for source lines"),
tools/perf/util/auxtrace.h +19 -0
@@ -576,6 +576,23 @@ static inline void auxtrace__free(struct perf_session *session)return session->auxtrace->free (session); } +#define ITRACE_HELP \ +" i: synthesize instructions events\n" \ +" b: synthesize branches events\n" \ +" c: synthesize branches events (calls only)\n" \ +" r: synthesize branches events (returns only)\n" \ +" x: synthesize transactions events\n" \ +" w: synthesize ptwrite events\n" \ +" p: synthesize power events\n" \ +" e: synthesize error events\n" \ +" d: create a debug log\n" \ +" g[len]: synthesize a call chain (use with i or x)\n" \ +" l[len]: synthesize last branch entries (use with i or x)\n" \ +" sNUMBER: skip initial number of events\n" \ +" PERIOD[ns|us|ms|i|t]: specify period to sample stream\n" \ +" concatenate multiple options. Default is ibxwpe or cewp\n" + +#else static inline struct auxtrace_record *
@@ -716,6 +733,8 @@ void auxtrace_mmap_params__set_idx(struct auxtrace_mmap_params *mp, struct perf_evlist *evlist, int idx, bool per_cpu); +#define ITRACE_HELP "" +#endif #endif
Signature · 1 line
2.17.1
PATCH v2 2/10 tools, pager: Support overwriting the pager
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add an interface to the auto pager code that allows callers to overwrite the pager.
--- tools/lib/subcmd/pager.c | 11 ++++++++++- tools/lib/subcmd/pager.h | 1 + 2 files changed, 11 insertions(+), 1 deletion(-)
tools/lib/subcmd/pager.c +10 -1
@@ -23,6 +23,13 @@ void pager_init(const char *pager_env) subcmd_config.pager_env = pager_env; } +static const char *forced_pager; + +void force_pager (const char *pager) +{ + forced_pager = pager; +} +static void pager_preexec (void ) {/*
@@ -66,7 +73,9 @@ void setup_pager(void)const char *pager =getenv (subcmd_config.pager_env);struct winsize sz; -if (!isatty (1 )) +if (forced_pager) + pager = forced_pager; +if (!isatty (1 ) && !forced_pager)return ;if (ioctl (1 ,TIOCGWINSZ , &sz) ==0 ) pager_columns = sz.ws_col;
tools/lib/subcmd/pager.h +1 -0
@@ -7,5 +7,6 @@ extern void pager_init(const char *pager_env);extern void setup_pager (void );extern int pager_in_use (void );extern int pager_get_columns (void ); +extern void force_pager (const char *);#endif /* __SUBCMD_PAGER_H */
Signature · 1 line
2.17.1
PATCH v2 3/10 tools, perf, script: Add --insn-trace for instruction decoding
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add a --insn-trace short hand option for decoding and disassembling instruction streams for intel_pt. This automatically pipes the output into the xed disassembler to generate disassembled instructions. This just makes this use model much nicer to use
Before
% perf record -e intel_pt// ...
% perf script --itrace=i0ns --ns -F +insn,-event,-period | xed -F insn: -A -64
swapper 0 [000] 117276.429606186: ffffffff81010486 pt_config ([kernel.kallsyms]) nopl %eax, (%rax,%rax,1) swapper 0 [000] 117276.429606186: ffffffff8101048b pt_config ([kernel.kallsyms]) add $0x10, %rsp swapper 0 [000] 117276.429606186: ffffffff8101048f pt_config ([kernel.kallsyms]) popq %rbx swapper 0 [000] 117276.429606186: ffffffff81010490 pt_config ([kernel.kallsyms]) popq %rbp swapper 0 [000] 117276.429606186: ffffffff81010491 pt_config ([kernel.kallsyms]) popq %r12 swapper 0 [000] 117276.429606186: ffffffff81010493 pt_config ([kernel.kallsyms]) popq %r13 swapper 0 [000] 117276.429606186: ffffffff81010495 pt_config ([kernel.kallsyms]) popq %r14 swapper 0 [000] 117276.429606186: ffffffff81010497 pt_config ([kernel.kallsyms]) popq %r15 swapper 0 [000] 117276.429606186: ffffffff81010499 pt_config ([kernel.kallsyms]) retq swapper 0 [000] 117276.429606186: ffffffff8101063e pt_event_add ([kernel.kallsyms]) cmpl $0x1, 0x1b0(%rbx) swapper 0 [000] 117276.429606186: ffffffff81010645 pt_event_add ([kernel.kallsyms]) mov $0xffffffea, %eax swapper 0 [000] 117276.429606186: ffffffff8101064a pt_event_add ([kernel.kallsyms]) mov $0x0, %edx swapper 0 [000] 117276.429606186: ffffffff8101064f pt_event_add ([kernel.kallsyms]) popq %rbx swapper 0 [000] 117276.429606186: ffffffff81010650 pt_event_add ([kernel.kallsyms]) cmovnz %edx, %eax swapper 0 [000] 117276.429606186: ffffffff81010653 pt_event_add ([kernel.kallsyms]) jmp 0xffffffff81010635 swapper 0 [000] 117276.429606186: ffffffff81010635 pt_event_add ([kernel.kallsyms]) retq swapper 0 [000] 117276.429606186: ffffffff8115e687 event_sched_in.isra.107 ([kernel.kallsyms]) test %eax, %eax
Now
% perf record -e intel_pt// ...
% perf script --insn-trace
... same output ...
--- tools/perf/Documentation/perf-script.txt | 4 ++++ tools/perf/builtin-script.c | 14 ++++++++++++++ 2 files changed, 18 insertions(+)
tools/perf/Documentation/perf-script.txt +4 -0
@@ -383,6 +383,10 @@ include::itrace.txt[] will be printed. Each entry has function name and file/line. Enabled by default, disable with --no-inline. +--insn-trace:: + Show instruction stream for intel_pt traces. Requires installing the + xed disassembler. + SEE ALSO -------- linkperf:perf-record[1], linkperf:perf-script-perl[1],
tools/perf/builtin-script.c +14 -0
@@ -44,6 +44,7 @@#include <sys/stat.h> #include <fcntl.h> #include <unistd.h> +#include <subcmd/pager.h> #include "sane_ctype.h"
@@ -3072,6 +3073,17 @@ static int perf_script__process_auxtrace_info(struct perf_tool *tool,#define perf_script__process_auxtrace_info 0#endif +static int parse_insn_trace (const struct option *opt__maybe_unused , +const char *str__maybe_unused , +int unset__maybe_unused ) +{ +parse_output_fields (NULL ,"+insn,-event,-period" ,0 ); +itrace_parse_synth_opts (opt,"i0ns" ,0 ); + nanosecs = true; +force_pager ("xed -F insn: -A -64 | less" ); +return 0 ; +} +int cmd_script (int argc,const char **argv) {bool show_full_info = false;
@@ -3156,6 +3168,8 @@ int cmd_script(int argc, const char **argv) "system-wide collection from all CPUs"),OPT_STRING ('S' ,"symbols" , &symbol_conf.sym_list_str,"symbol[,symbol...]" ,"only consider these symbols" ), +OPT_CALLBACK_OPTARG (0 ,"insn-trace" , &itrace_synth_opts,NULL ,NULL , +"Decode instructions from itrace" , parse_insn_trace),OPT_STRING (0 ,"stop-bt" , &symbol_conf.bt_stop_list_str,"symbol[,symbol...]" ,"Stop display of callgraph at these symbols" ), OPT_STRING('C', "cpu", &cpu_list, "cpu", "list of cpus to profile"),
Signature · 1 line
2.17.1
PATCH v2 4/10 perf, tools, script: Allow sym and dso without ip, addr
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Currently sym and dso require printing ip and addr because the print function is tied to those outputs. With callindent it makes sense to print the symbol or dso without numerical IP or ADDR. So change the dependency check to only check the underlying attribute.
This allows to remove a lot of noise in callindent output by dropping the numerical addresses, which are not all that useful.
Before
% perf script --itrace=cr -F +callindent
swapper 0 [000] 156546.354971: 1 branches: pt_config 0 [unknown] ([unknown]) => ffffffff81010486 pt_config ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: pt_config ffffffff81010499 pt_config ([kernel.kallsyms]) => ffffffff8101063e pt_event_add ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: pt_event_add ffffffff81010635 pt_event_add ([kernel.kallsyms]) => ffffffff8115e687 event_sched_in.isra.107 ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable ffffffff8115e726 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff811579b0 perf_pmu_enable ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void ffffffff81151730 perf_pmu_nop_void ([kernel.kallsyms]) => ffffffff8115e72b event_sched_in.isra.107 ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107 ffffffff8115e737 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff8115e7a5 group_sched_in ([kernel.kallsyms])
swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax ffffffff8115e7f6 group_sched_in ([kernel.kallsyms]) => ffffffff81a03000 __x86_indirect_thunk_rax ([kernel.kallsyms])After
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff
swapper 0 [000] 156546.354971: 1 branches: pt_config
swapper 0 [000] 156546.354971: 1 branches: pt_config
swapper 0 [000] 156546.354971: 1 branches: pt_event_add
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void
swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107
swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_int
swapper 0 [000] 156546.354971: 1 branches: group_sched_in
swapper 0 [000] 156546.354971: 1 branches: event_filter_match
swapper 0 [000] 156546.354971: 1 branches: event_filter_match
swapper 0 [000] 156546.354971: 1 branches: group_sched_in--- tools/perf/builtin-script.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
tools/perf/builtin-script.c +6 -6
@@ -407,9 +407,10 @@ static int perf_evsel__check_attr(struct perf_evsel *evsel, PERF_OUTPUT_WEIGHT))return -EINVAL ; -if (PRINT_FIELD (SYM ) && !PRINT_FIELD (IP ) && !PRINT_FIELD (ADDR )) { +if (PRINT_FIELD (SYM ) && + !(evsel->attr.sample_type & (PERF_SAMPLE_IP |PERF_SAMPLE_ADDR ))) {pr_err ("Display of symbols requested but neither sample IP nor " -"sample address\nis selected. Hence, no addresses to convert " +"sample address\navailable. Hence, no addresses to convert " "to symbols.\n" );return -EINVAL ; }
@@ -418,10 +419,9 @@ static int perf_evsel__check_attr(struct perf_evsel *evsel, "selected.\n");return -EINVAL ; } -if (PRINT_FIELD (DSO ) && !PRINT_FIELD (IP ) && !PRINT_FIELD (ADDR ) && - !PRINT_FIELD (BRSTACK ) && !PRINT_FIELD (BRSTACKSYM ) && !PRINT_FIELD (BRSTACKOFF )) { -pr_err ("Display of DSO requested but no address to convert. Select\n" -"sample IP, sample address, brstack, brstacksym, or brstackoff.\n" ); +if (PRINT_FIELD (DSO ) && + !(evsel->attr.sample_type & (PERF_SAMPLE_IP |PERF_SAMPLE_ADDR ))) { +pr_err ("Display of DSO requested but no address to convert.\n" );return -EINVAL ; }if (PRINT_FIELD (SRCLINE ) && !PRINT_FIELD (IP )) {
Signature · 1 line
2.17.1
Re: [PATCH v2 04/10] perf, tools, script: Allow sym and dso without ip, addr
Quoted 25 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Currently sym and dso require printing ip and addr because the print function is tied to those outputs. With callindent it makes sense to print the symbol or dso without numerical IP or ADDR. So change the dependency check to only check the underlying attribute.
This allows to remove a lot of noise in callindent output by dropping the numerical addresses, which are not all that useful.
Before
% perf script --itrace=cr -F +callindent
swapper 0 [000] 156546.354971: 1 branches: pt_config 0 [unknown] ([unknown]) => ffffffff81010486 pt_config ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: pt_config ffffffff81010499 pt_config ([kernel.kallsyms]) => ffffffff8101063e pt_event_add ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: pt_event_add ffffffff81010635 pt_event_add ([kernel.kallsyms]) => ffffffff8115e687 event_sched_in.isra.107 ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable ffffffff8115e726 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff811579b0 perf_pmu_enable ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void ffffffff81151730 perf_pmu_nop_void ([kernel.kallsyms]) => ffffffff8115e72b event_sched_in.isra.107 ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107 ffffffff8115e737 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff8115e7a5 group_sched_in ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax ffffffff8115e7f6 group_sched_in ([kernel.kallsyms]) => ffffffff81a03000 __x86_indirect_thunk_rax ([kernel.kallsyms])After
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff
Quoted 12 lines
swapper 0 [000] 156546.354971: 1 branches: pt_config
swapper 0 [000] 156546.354971: 1 branches: pt_config
swapper 0 [000] 156546.354971: 1 branches: pt_event_add
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void
swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107
swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax
swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_int
swapper 0 [000] 156546.354971: 1 branches: group_sched_in
swapper 0 [000] 156546.354971: 1 branches: event_filter_match
swapper 0 [000] 156546.354971: 1 branches: event_filter_match
swapper 0 [000] 156546.354971: 1 branches: group_sched_inI'm getting different output than advertised
[jolsa@krava perf]$ ./perf script --itrace=cr -F +callindent,-ip,-sym,-symoff | head -3
ls 1926 [000] 55509.544711: 1 branches:uhH: _start => 7f9722fa0f60 (/usr/lib64/ld-2.27.so)
ls 1926 [000] 55509.544711: 1 branches:uhH: _start => 0 ([unknown])
ls 1926 [000] 55509.544716: 1 branches:uhH: _start => 7f9722fa0f60 (/usr/lib64/ld-2.27.so)jirka
Re: [PATCH v2 04/10] perf, tools, script: Allow sym and dso without ip, addr
Quoted 39 lines · Jiri Olsa
Quoted 25 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Currently sym and dso require printing ip and addr because the print function is tied to those outputs. With callindent it makes sense to print the symbol or dso without numerical IP or ADDR. So change the dependency check to only check the underlying attribute.
This allows to remove a lot of noise in callindent output by dropping the numerical addresses, which are not all that useful.
Before
% perf script --itrace=cr -F +callindent
swapper 0 [000] 156546.354971: 1 branches: pt_config 0 [unknown] ([unknown]) => ffffffff81010486 pt_config ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: pt_config ffffffff81010499 pt_config ([kernel.kallsyms]) => ffffffff8101063e pt_event_add ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: pt_event_add ffffffff81010635 pt_event_add ([kernel.kallsyms]) => ffffffff8115e687 event_sched_in.isra.107 ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable ffffffff8115e726 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff811579b0 perf_pmu_enable ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void ffffffff81151730 perf_pmu_nop_void ([kernel.kallsyms]) => ffffffff8115e72b event_sched_in.isra.107 ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107 ffffffff8115e737 event_sched_in.isra.107 ([kernel.kallsyms]) => ffffffff8115e7a5 group_sched_in ([kernel.kallsyms]) swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax ffffffff8115e7f6 group_sched_in ([kernel.kallsyms]) => ffffffff81a03000 __x86_indirect_thunk_rax ([kernel.kallsyms])After
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff
Quoted 12 lines
swapper 0 [000] 156546.354971: 1 branches: pt_config swapper 0 [000] 156546.354971: 1 branches: pt_config swapper 0 [000] 156546.354971: 1 branches: pt_event_add swapper 0 [000] 156546.354971: 1 branches: perf_pmu_enable swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_void swapper 0 [000] 156546.354971: 1 branches: event_sched_in.isra.107 swapper 0 [000] 156546.354971: 1 branches: __x86_indirect_thunk_rax swapper 0 [000] 156546.354971: 1 branches: perf_pmu_nop_int swapper 0 [000] 156546.354971: 1 branches: group_sched_in swapper 0 [000] 156546.354971: 1 branches: event_filter_match swapper 0 [000] 156546.354971: 1 branches: event_filter_match swapper 0 [000] 156546.354971: 1 branches: group_sched_in
Quoted 6 lines
I'm getting different output than advertised
[jolsa@krava perf]$ ./perf script --itrace=cr -F +callindent,-ip,-sym,-symoff | head -3
ls 1926 [000] 55509.544711: 1 branches:uhH: _start => 7f9722fa0f60 (/usr/lib64/ld-2.27.so) ls 1926 [000] 55509.544711: 1 branches:uhH: _start => 0 ([unknown]) ls 1926 [000] 55509.544716: 1 branches:uhH: _start => 7f9722fa0f60 (/usr/lib64/ld-2.27.so)
Yes it needs the next patch too. There's no good way to show the effect independently.
I will merge them into one.
-Andi
PATCH v2 5/10 tools, perf, script: Set user_set/wildcard_set for +- fields in -F
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
The branch target output relies on the user_set flag to determine if the branch target should be implicitely printed. When modifying the fields with + or - also set user_set,
so that ADDR can be removed. We also need to set wildcard_set to make the initial sanity check pass.
Before:
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff,-addr
swapper 0 [000] 3377.917072: 1 branches: pt_config => ffffffff81010486 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: pt_config => ffffffff8101063e ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: pt_event_add => ffffffff8115e687 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_enable => ffffffff811579b0 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_void => ffffffff8115e72b ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: event_sched_in.isra.107 => ffffffff8115e7a5 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: __x86_indirect_thunk_rax => ffffffff81a03000 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_int => ffffffff8115e7fb ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: group_sched_in => ffffffff811583a8 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: event_filter_match => ffffffff81153e10 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: event_filter_match => ffffffff811583d4 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: group_sched_in => ffffffff8115e738 ([kernel.kallsyms])
swapper 0 [000] 3377.917072: 1 branches: __x86_indirect_thunk_rax => ffffffff81a03000 ([kernel.kallsyms])After:
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff,-addr
swapper 0 [000] 3377.917072: 1 branches: pt_config
swapper 0 [000] 3377.917072: 1 branches: pt_config
swapper 0 [000] 3377.917072: 1 branches: pt_event_add
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_enable
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_void
swapper 0 [000] 3377.917072: 1 branches: event_sched_in.isra.107
swapper 0 [000] 3377.917072: 1 branches: __x86_indirect_thunk_rax
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_int
swapper 0 [000] 3377.917072: 1 branches: group_sched_in
swapper 0 [000] 3377.917072: 1 branches: event_filter_match
swapper 0 [000] 3377.917072: 1 branches: event_filter_match
swapper 0 [000] 3377.917072: 1 branches: group_sched_inv2: Use correct index for fallback case
--- tools/perf/builtin-script.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
tools/perf/builtin-script.c +4 -1
@@ -2490,6 +2490,8 @@ static int parse_output_fields(const struct option *opt __maybe_unused, output[j].fields &= ~all_output_options[i].field; else output[j].fields |= all_output_options[i].field; + output[j].user_set = true; + output[j].wildcard_set = true; } } }else {
@@ -2500,7 +2502,8 @@ static int parse_output_fields(const struct option *opt __maybe_unused, rc = -EINVAL ; goto out; } - output[type].fields |= all_output_options[i].field; + output[type].user_set = true; + output[type].wildcard_set = true; } }
Signature · 1 line
2.17.1
PATCH v2 6/10 perf, tools, script: Implement - for typed fields
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
For perf script -F the - removal of fields didn't work for typed fields, only for untyped. Add the REMOVE case for this variant too.
v2: Use correct index
--- tools/perf/builtin-script.c | 4 ++++ 1 file changed, 4 insertions(+)
tools/perf/builtin-script.c +4 -0
@@ -2502,6 +2502,10 @@ static int parse_output_fields(const struct option *opt __maybe_unused, rc = -EINVAL ; goto out; } +if (change ==REMOVE ) + output[type].fields &= ~all_output_options[i].field; +else + output[type].fields |= all_output_options[i].field; output[type].user_set = true; output[type].wildcard_set = true; }
Signature · 1 line
2.17.1
Re: [PATCH v2 06/10] perf, tools, script: Implement - for typed fields
Quoted 4 lines · Andi Kleen
From: Andi Kleen <[email protected]>
For perf script -F the - removal of fields didn't work for typed fields, only for untyped. Add the REMOVE case for this variant too.
missing changelog example
jirka
Quoted 25 lines
v2: Use correct index
Signed-off-by Andi Kleen <[email protected]>--- tools/perf/builtin-script.c | 4 ++++ 1 file changed, 4 insertions(+)tools/perf/builtin-script.c +4
-0@@ -2502,6 +2502,10 @@ static int parse_output_fields(const struct option *opt __maybe_unused, rc = -EINVAL ; goto out; } +if (change ==REMOVE ) + output[type].fields &= ~all_output_options[i].field; +else + output[type].fields |= all_output_options[i].field; output[type].user_set = true; output[type].wildcard_set = true; }Signature · 1 line
2.17.1
PATCH v2 7/10 perf, tools, script: Print DSO for callindent
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Now that we don't need to print the IP/ADDR for callindent the DSO is also not printed. It's useful for some cases, so add an own DSO printout for callindent for the case when IP/ADDR is not enabled.
Before:
% perf script --itrace=cr -F +callindent,-ip,-sym,-symoff,-addr
swapper 0 [000] 3377.917072: 1 branches: pt_config
swapper 0 [000] 3377.917072: 1 branches: pt_config
swapper 0 [000] 3377.917072: 1 branches: pt_event_add
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_enable
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_void
swapper 0 [000] 3377.917072: 1 branches: event_sched_in.isra.107
swapper 0 [000] 3377.917072: 1 branches: __x86_indirect_thunk_rax
swapper 0 [000] 3377.917072: 1 branches: perf_pmu_nop_int
swapper 0 [000] 3377.917072: 1 branches: group_sched_in
swapper 0 [000] 3377.917072: 1 branches: event_filter_match
swapper 0 [000] 3377.917072: 1 branches: event_filter_match
swapper 0 [000] 3377.917072: 1 branches: group_sched_inAfter:
swapper 0 [000] 3377.917072: 1 branches: ([unknown]) pt_config
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) pt_config
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) pt_event_add
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) perf_pmu_enable
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) perf_pmu_nop_void
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) event_sched_in.isra.107
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) perf_pmu_nop_int
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) group_sched_in
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) event_filter_match
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) event_filter_match
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) group_sched_in
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) perf_pmu_nop_txn
swapper 0 [000] 3377.917072: 1 branches: ([kernel.kallsyms]) event_sched_in.isra.107(in the kernel case of course it's not very useful, but it's important with user programs where symbols are not unique)
--- tools/perf/builtin-script.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
tools/perf/builtin-script.c +8 -1
@@ -1116,6 +1116,7 @@ static int perf_sample__fprintf_callindent(struct perf_sample *sample,const char *name =NULL ;static int spacing;int len =0 ; +int dlen =0 ;u64 ip =0 ;/*
@@ -1142,6 +1143,12 @@ static int perf_sample__fprintf_callindent(struct perf_sample *sample, ip = sample->ip; } +if (PRINT_FIELD (DSO ) && !(PRINT_FIELD (IP ) ||PRINT_FIELD (ADDR ))) { + dlen +=fprintf (fp,"(" ); + dlen +=map__fprintf_dsoname (al->map, fp); + dlen +=fprintf (fp,")\t" ); + } +if (name) len =fprintf (fp,"%*s%s" , (int )depth *4 ,"" , name);else if (ip)
@@ -1160,7 +1167,7 @@ static int perf_sample__fprintf_callindent(struct perf_sample *sample,if (len < spacing) len +=fprintf (fp,"%*s" , spacing - len,"" ); -return len; +return len + dlen; } static int perf_sample__fprintf_insn(struct perf_sample *sample,
Signature · 1 line
2.17.1
PATCH v2 8/10 perf, tools, script: Make itrace script default to all calls
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
By default perf script for itrace outputs sampled instructions or branches. In my experience this is confusing to users because it's hard to correlate with real program behavior. The sampling makes sense for tools like report that actually sample to reduce the run time, but run time is normally not a problem for perf script. It's better to give an accurate representation of the program flow.
Default perf script to output all calls for itrace. That's a much saner default. The old behavior can be still requested with
perf script --itrace=ibxwpe100000
--- tools/perf/Documentation/itrace.txt | 7 ++++--- tools/perf/builtin-script.c | 5 ++++- tools/perf/util/auxtrace.c | 17 ++++++++++++----- tools/perf/util/auxtrace.h | 5 ++++- tools/perf/util/cs-etm.c | 3 ++- tools/perf/util/intel-bts.c | 3 ++- tools/perf/util/intel-pt.c | 3 ++- 7 files changed, 30 insertions(+), 13 deletions(-)
tools/perf/Documentation/itrace.txt +4 -3
@@ -11,10 +11,11 @@ l synthesize last branch entries (use with i or x) s skip initial number of events - The default is all events i.e. the same as --itrace=ibxwpe + The default is all events i.e. the same as --itrace=ibxwpe, + except for perf script where it is --itrace=ce - In addition, the period (default 100000) for instructions events - can be specified in units of: + In addition, the period (default 100000, except for perf script where it is 1) + for instructions events can be specified in units of: i instructions t ticks
tools/perf/builtin-script.c +4 -1
@@ -3107,7 +3107,10 @@ int cmd_script(int argc, const char **argv)char *rec_script_path =NULL ;char *rep_script_path =NULL ;struct perf_session *session; -struct itrace_synth_opts itrace_synth_opts = { .set = false, }; +struct itrace_synth_opts itrace_synth_opts = { + .set = false, + .default_no_sample = true, + };char *script_path =NULL ;const char **__argv;int i, j, err =0 ;
tools/perf/util/auxtrace.c +12 -5
@@ -958,16 +958,23 @@ s64 perf_event__process_auxtrace(struct perf_tool *tool,#define PERF_ITRACE_DEFAULT_LAST_BRANCH_SZ 64#define PERF_ITRACE_MAX_LAST_BRANCH_SZ 1024 -void itrace_synth_opts__set_default (struct itrace_synth_opts *synth_opts) +void itrace_synth_opts__set_default (struct itrace_synth_opts *synth_opts, +bool no_sample) { - synth_opts->instructions = true; synth_opts->branches = true; synth_opts->transactions = true; synth_opts->ptwrites = true; synth_opts->pwr_events = true; synth_opts->errors = true; - synth_opts->period_type =PERF_ITRACE_DEFAULT_PERIOD_TYPE ; - synth_opts->period =PERF_ITRACE_DEFAULT_PERIOD ; +if (no_sample) { + synth_opts->period_type =PERF_ITRACE_PERIOD_INSTRUCTIONS ; + synth_opts->period =1 ; + synth_opts->calls = true; + }else { + synth_opts->instructions = true; + synth_opts->period_type =PERF_ITRACE_DEFAULT_PERIOD_TYPE ; + synth_opts->period =PERF_ITRACE_DEFAULT_PERIOD ; + } synth_opts->callchain_sz =PERF_ITRACE_DEFAULT_CALLCHAIN_SZ ; synth_opts->last_branch_sz =PERF_ITRACE_DEFAULT_LAST_BRANCH_SZ ; synth_opts->initial_skip =0 ;
@@ -995,7 +1002,7 @@ int itrace_parse_synth_opts(const struct option *opt, const char *str, }if (!str) { -itrace_synth_opts__set_default (synth_opts); +itrace_synth_opts__set_default (synth_opts, false);return 0 ; }
tools/perf/util/auxtrace.h +4 -1
@@ -55,6 +55,7 @@ enum itrace_period_type {/** * struct itrace_synth_opts - AUX area tracing synthesis options. * @set: indicates whether or not options have been set +* @default_no_sample: Default to no sampling. * @inject: indicates the event (not just the sample) must be fully synthesized * because 'perf inject' will write it out * @instructions: whether to synthesize 'instructions' events
@@ -79,6 +80,7 @@ enum itrace_period_type {*/ struct itrace_synth_opts {bool set; +bool default_no_sample;bool inject;bool instructions;bool branches;
@@ -527,7 +529,8 @@ int perf_event__process_auxtrace_error(struct perf_tool *tool, struct perf_session *session);int itrace_parse_synth_opts (const struct option *opt,const char *str,int unset); -void itrace_synth_opts__set_default (struct itrace_synth_opts *synth_opts); +void itrace_synth_opts__set_default (struct itrace_synth_opts *synth_opts, +bool no_sample);size_t perf_event__fprintf_auxtrace_error (union perf_event *event,FILE *fp); void perf_session__auxtrace_error_inc(struct perf_session *session,
tools/perf/util/cs-etm.c +2 -1
@@ -1432,7 +1432,8 @@ int cs_etm__process_auxtrace_info(union perf_event *event,if (session->itrace_synth_opts && session->itrace_synth_opts->set) { etm->synth_opts = *session->itrace_synth_opts; }else { -itrace_synth_opts__set_default (&etm->synth_opts); +itrace_synth_opts__set_default (&etm->synth_opts, + session->itrace_synth.default_no_sample); etm->synth_opts.callchain = false; }
tools/perf/util/intel-bts.c +2 -1
@@ -910,7 +910,8 @@ int intel_bts_process_auxtrace_info(union perf_event *event,if (session->itrace_synth_opts && session->itrace_synth_opts->set) { bts->synth_opts = *session->itrace_synth_opts; }else { -itrace_synth_opts__set_default (&bts->synth_opts); +itrace_synth_opts__set_default (&bts->synth_opts, + session->itrace_synth_opts->default_no_sample);if (session->itrace_synth_opts) bts->synth_opts.thread_stack = session->itrace_synth_opts->thread_stack;
tools/perf/util/intel-pt.c +2 -1
@@ -2554,7 +2554,8 @@ int intel_pt_process_auxtrace_info(union perf_event *event,if (session->itrace_synth_opts && session->itrace_synth_opts->set) { pt->synth_opts = *session->itrace_synth_opts; }else { -itrace_synth_opts__set_default (&pt->synth_opts); +itrace_synth_opts__set_default (&pt->synth_opts, + session->itrace_synth_opts->default_no_sample);if (use_browser !=-1 ) { pt->synth_opts.branches = false; pt->synth_opts.callchain = true;
Signature · 1 line
2.17.1
PATCH v2 9/10 tools, perf, script: Add --call-trace and --call-ret-trace
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add short cut options to print PT call trace and call-ret-trace, for calls and call and returns. Roughly corresponds to ftrace function tracer and function graph tracer.
Just makes these common use cases nicer to use.
% perf record -a -e intel_pt// sleep 1
% perf script --call-trace
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpage% perf script --call-ret-trace
perf 900 [000] 194167.205652203: tr strt ([unknown]) pt_config
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) pt_config
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) pt_event_add
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) perf_pmu_nop_void
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) perf_pmu_nop_int
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) event_filter_match
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: return ([kernel.kallsyms]) perf_pmu_nop_txn
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: call ([kernel.kallsyms]) perf_event_set_state.part.71v2: Print errors, power, ptwrite too
--- tools/perf/Documentation/perf-script.txt | 7 +++++++ tools/perf/builtin-script.c | 24 ++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
tools/perf/Documentation/perf-script.txt +7 -0
@@ -387,6 +387,13 @@ include::itrace.txt[] Show instruction stream for intel_pt traces. Requires installing the xed disassembler. +--call-trace:: + Show call stream for intel_pt traces. The CPUs are interleaved, but + can be filtered with -C. + +--call-ret-trace:: + Show call and return stream for intel_pt traces. + SEE ALSO -------- linkperf:perf-record[1], linkperf:perf-script-perl[1],
tools/perf/builtin-script.c +24 -0
@@ -3098,6 +3098,26 @@ static int parse_insn_trace(const struct option *opt __maybe_unused,return 0 ; } +static int parse_call_trace (const struct option *opt__maybe_unused , +const char *str__maybe_unused , +int unset__maybe_unused ) +{ +parse_output_fields (NULL ,"-ip,-addr,-event,-period,+callindent" ,0 ); +itrace_parse_synth_opts (opt,"cewp" ,0 ); + nanosecs = true; +return 0 ; +} + +static int parse_callret_trace (const struct option *opt__maybe_unused , +const char *str__maybe_unused , +int unset__maybe_unused ) +{ +parse_output_fields (NULL ,"-ip,-addr,-event,-period,+callindent,+flags" ,0 ); +itrace_parse_synth_opts (opt,"crewp" ,0 ); + nanosecs = true; +return 0 ; +} +int cmd_script (int argc,const char **argv) {bool show_full_info = false;
@@ -3187,6 +3207,10 @@ int cmd_script(int argc, const char **argv) "only consider these symbols"),OPT_CALLBACK_OPTARG (0 ,"insn-trace" , &itrace_synth_opts,NULL ,NULL ,"Decode instructions from itrace" , parse_insn_trace), +OPT_CALLBACK_OPTARG (0 ,"call-trace" , &itrace_synth_opts,NULL ,NULL , +"Decode calls from from itrace" , parse_call_trace), +OPT_CALLBACK_OPTARG (0 ,"call-ret-trace" , &itrace_synth_opts,NULL ,NULL , +"Decode calls and returns from itrace" , parse_callret_trace),OPT_STRING (0 ,"stop-bt" , &symbol_conf.bt_stop_list_str,"symbol[,symbol...]" ,"Stop display of callgraph at these symbols" ), OPT_STRING('C', "cpu", &cpu_list, "cpu", "list of cpus to profile"),
Signature · 1 line
2.17.1
PATCH v2 10/10 tools, perf, script: Implement --graph-function
In reply to Andi KleenFrom: Andi Kleen <[email protected]>
Add a ftrace style --graph-function argument to perf script that allows to print itrace function calls only below a given function. This makes it easier to find the code of interest in a large trace.
% perf record -e intel_pt//k -a sleep 1
% perf script --graph-function group_sched_in --call-trace
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpage
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) calc_timer_values
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_cpu
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) arch_perf_update_userpage
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __fentry__
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) using_native_sched_clock
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_stable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) group_sched_in
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) event_sched_in.isra.107
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_set_state.part.71
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_time
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_pmu_disable
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_log_itrace_start
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_userpage
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) calc_timer_values
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_cpu
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) arch_perf_update_userpage
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __fentry__
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) using_native_sched_clock
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_stable--- tools/perf/Documentation/perf-script.txt | 4 + tools/perf/builtin-script.c | 97 +++++++++++++++++++----- tools/perf/util/symbol.h | 3 +- tools/perf/util/thread.h | 2 + 4 files changed, 87 insertions(+), 19 deletions(-)
tools/perf/Documentation/perf-script.txt +4 -0
@@ -394,6 +394,10 @@ include::itrace.txt[] --call-ret-trace:: Show call and return stream for intel_pt traces. +--graph-function:: + For itrace only show specified functions and their callees for + itrace. Multiple functions can be separated by comma. + SEE ALSO -------- linkperf:perf-record[1], linkperf:perf-script-perl[1],
tools/perf/builtin-script.c +79 -18
145 lines not on this page · raw message
tools/perf/util/symbol.h +2 -1
@@ -123,7 +123,8 @@ struct symbol_conf {const char *vmlinux_name, *kallsyms_name, *source_prefix, - *field_sep; + *field_sep, + *graph_function;const char *default_guest_vmlinux_name, *default_guest_kallsyms, *default_guest_modules;
tools/perf/util/thread.h +2 -0
@@ -42,6 +42,8 @@ struct thread {void *addr_space;struct unwind_libunwind_ops *unwind_libunwind_ops;#endif +bool filter; +int filter_entry_depth; };struct machine ;
Signature · 1 line
2.17.1
Re: [PATCH v2 10/10] tools, perf, script: Implement --graph-function
Quoted 34 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Add a ftrace style --graph-function argument to perf script that allows to print itrace function calls only below a given function. This makes it easier to find the code of interest in a large trace.
% perf record -e intel_pt//k -a sleep 1
% perf script --graph-functiongroup_sched_in--call-traceperf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107 perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71 perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpage perf 900 [000] 194167.205652203: ([kernel.kallsyms]) calc_timer_values perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_cpu perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) arch_perf_update_userpage perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __fentry__ perf 900 [000] 194167.205652203: ([kernel.kallsyms]) using_native_sched_clock perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_stable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) group_sched_in swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) event_sched_in.isra.107 swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_set_state.part.71 swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_time swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_pmu_disable swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_log_itrace_start swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax
Quoted 8 lines
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_userpage
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) calc_timer_values
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_cpu
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) arch_perf_update_userpage
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __fentry__
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) using_native_sched_clock
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_stablenice, but I'm also getting a lot of these lines:
instruction trace error type 1 cpu 0 pid 0 tid 0 ip 0xffffffffb3a017a0 code 6: Trace doesn't match instruction instruction trace error type 1 cpu 0 pid 0 tid 0 ip 0xffffffffb3a017a0 code 6: Trace doesn't match instruction instruction trace error type 1 cpu 0 pid 0 tid 0 ip 0xffffffffb3a017a0 code 6: Trace doesn't match instruction
any idea where are those comming from?
thanks,
jirka
Re: [PATCH v2 10/10] tools, perf, script: Implement --graph-function
Quoted 44 lines · Jiri Olsa
Quoted 34 lines · Andi Kleen
From: Andi Kleen <[email protected]>
Add a ftrace style --graph-function argument to perf script that allows to print itrace function calls only below a given function. This makes it easier to find the code of interest in a large trace.
% perf record -e intel_pt//k -a sleep 1
% perf script --graph-functiongroup_sched_in--call-traceperf 900 [000] 194167.205652203: ([kernel.kallsyms]) group_sched_in perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) event_sched_in.isra.107 perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_set_state.part.71 perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_time perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_disable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_log_itrace_start perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_event_update_userpage perf 900 [000] 194167.205652203: ([kernel.kallsyms]) calc_timer_values perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_cpu perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax perf 900 [000] 194167.205652203: ([kernel.kallsyms]) arch_perf_update_userpage perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __fentry__ perf 900 [000] 194167.205652203: ([kernel.kallsyms]) using_native_sched_clock perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_stable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) group_sched_in swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) event_sched_in.isra.107 swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_set_state.part.71 swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_time swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_pmu_disable swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_log_itrace_start swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_raxQuoted 8 lines
swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) perf_event_update_userpage swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) calc_timer_values swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_cpu swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __x86_indirect_thunk_rax swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) arch_perf_update_userpage swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) __fentry__ swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) using_native_sched_clock swapper 0 [001] 194167.205660693: ([kernel.kallsyms]) sched_clock_stable
Quoted 7 lines
nice, but I'm also getting a lot of these lines:
instruction trace error type 1 cpu 0 pid 0 tid 0 ip
0xffffffffb3a017a0code 6: Trace doesn't match instruction instruction trace error type 1 cpu 0 pid 0 tid 0 ip0xffffffffb3a017a0code 6: Trace doesn't match instruction instruction trace error type 1 cpu 0 pid 0 tid 0 ip0xffffffffb3a017a0code 6: Trace doesn't match instructionany idea where are those comming from?
PT decoder didn't see the same instructions as during the trace collection.
There are many reasons which can cause this, usually (but not always) some form of self modifying code. The Linux kernel does a lot of self modification these days, which is a challenge for PT.
If you run the decoder as root it will decode from /proc/kcore, which helps, but only if no jump label has changed state since the collection.
The perf PT decoder tends to resync relatively quickly, so usually not too much is lost.
-Andi
Quoted 5 lines · On Fri, 31 Aug 2018 15:01:56 -0700 Andi Kleen <[email protected]> wrote:
Also available in
git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc.gitpt/easy-script-2v1: Initial post
v2: Address review comments. Minor fixes to descriptions. Now builds everywhere.
OK, I grabbed the pt/easy-script-2 branch (commit 4c69fd388e17), and see build failures for a 'make -C perf' build using gcc version 7.3.0 (Ubuntu 18.04):
builtin-script.c: In function ‘parse_output_fields’:
builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].fields &= ~all_output_options[i].field;
^~
builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].fields &= ~all_output_options[i].field;
~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
builtin-script.c:2574:10: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].user_set = true;
~~~~~~^~~
builtin-script.c:2575:10: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].wildcard_set = true;
~~~~~~^~~
...
util/cs-etm.c: In function ‘cs_etm__process_auxtrace_info’:
util/cs-etm.c:1436:14: error: ‘struct perf_session’ has no member named ‘itrace_synth’; did you mean ‘itrace_synth_opts’?
session->itrace_synth.default_no_sample);
^~~~~~~~~~~~
itrace_synth_optsThanks,
Kim
p.s. FWIW, my make VF=1 output:
Auto-detecting system features:
... dwarf: [ on ] ... dwarf_getlocations: [ on ] ... glibc: [ on ] ... gtk2: [ on ] ... libaudit: [ on ] ... libbfd: [ on ] ... libelf: [ on ] ... libnuma: [ on ] ... numa_num_possible_cpus: [ on ] ... libperl: [ on ] ... libpython: [ on ] ... libslang: [ on ] ... libcrypto: [ on ] ... libunwind: [ on ] ... libdw-dwarf-unwind: [ on ] ... zlib: [ on ] ... lzma: [ on ] ... get_cpuid: [ on ] ... bpf: [ on ] ... backtrace: [ on ] ... fortify-source: [ on ] ... sync-compare-and-swap: [ on ] ... gtk2-infobar: [ on ] ... libelf-getphdrnum: [ on ] ... libelf-gelf_getnote: [ on ] ... libelf-getshdrstrndx: [ on ] ... libelf-mmap: [ on ] ... libpython-version: [ on ]
... libunwind-x86: [ OFF ]
... libunwind-x86_64: [ on ]
... libunwind-arm: [ OFF ]
... libunwind-aarch64: [ OFF ]
... pthread-attr-setaffinity-np: [ on ] ... pthread-barrier: [ on ] ... stackprotector-all: [ on ] ... timerfd: [ on ] ... sched_getcpu: [ on ] ... sdt: [ on ] ... setns: [ on ] ... libopencsd: [ on ]
... bionic: [ OFF ]
... compile-32: [ OFF ]
... compile-x32: [ OFF ]
... cplus-demangle: [ on ]
... hello: [ OFF ]
... libbabeltrace: [ on ] ... liberty: [ on ] ... liberty-z: [ on ]
... libunwind-debug-frame: [ OFF ]
... libunwind-debug-frame-arm: [ OFF ]
... libunwind-debug-frame-aarch64: [ OFF ]
... cxx: [ OFF ]
... llvm: [ OFF ]
... llvm-version: [ OFF ]
... clang: [ OFF ]
... prefix: /home/kim ... bindir: /home/kim/bin ... libdir: /home/kim/lib64
... sysconfdir: /home/kim/etc
... LIBUNWIND_DIR:
... LIBDW_DIR:
... JDIR: /usr/lib/jvm/java-1.8.0-openjdk-amd64
... DWARF post unwind library: libunwind
Quoted 24 lines · Kim Phillips
Quoted 5 lines · On Fri, 31 Aug 2018 15:01:56 -0700 Andi Kleen <[email protected]> wrote:
Also available in
git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc.gitpt/easy-script-2v1: Initial post
v2: Address review comments. Minor fixes to descriptions. Now builds everywhere.OK, I grabbed the pt/easy-script-2 branch (commit
4c69fd388e17), and see build failures for a 'make -C perf' build using gcc version 7.3.0 (Ubuntu 18.04):
builtin-script.c: In function ‘parse_output_fields’:
builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds]output[j].fields &= ~all_output_options[i].field; ^~ builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds] output[j].fields &= ~all_output_options[i].field; ~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ builtin-script.c:2574:10: error: array subscript is above array bounds [-Werror=array-bounds] output[j].user_set = true; ~~~~~~^~~builtin-script.c:2575:10: error: array subscript is above array bounds [-Werror=array-bounds]
Quoted 8 lines
output[j].wildcard_set = true;
~~~~~~^~~
...
util/cs-etm.c: In function ‘cs_etm__process_auxtrace_info’:
util/cs-etm.c:1436:14: error: ‘struct perf_session’ has no member named ‘itrace_synth’; did you mean ‘itrace_synth_opts’?
session->itrace_synth.default_no_sample);
^~~~~~~~~~~~
itrace_synth_optsI think Andi has the new changes in pt/easy-script-3 branch, just did not update the changelog
jirka
Quoted 4 lines
(Ubuntu 18.04):
builtin-script.c: In function ‘parse_output_fields’:
builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds]
Quoted 11 lines
output[j].fields &= ~all_output_options[i].field;
^~
builtin-script.c:2571:22: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].fields &= ~all_output_options[i].field;
~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
builtin-script.c:2574:10: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].user_set = true;
~~~~~~^~~
builtin-script.c:2575:10: error: array subscript is above array bounds [-Werror=array-bounds]
output[j].wildcard_set = true;
~~~~~~^~~These were fixed earlier in -3.
Quoted 6 lines
...
util/cs-etm.c: In function ‘cs_etm__process_auxtrace_info’:util/cs-etm.c:1436:14: error: ‘struct perf_session’ has no member named ‘itrace_synth’; did you mean ‘itrace_synth_opts’?session->itrace_synth.default_no_sample); ^~~~~~~~~~~~ itrace_synth_opts
For some reason my x86 targetted build doesn't seem to cover this file, so I missed it. What do I need to do to enable it?
-Andi
Jiri Olsa
Acked-byOn Fri, Aug 31, 2018 at 03:01:56PM -0700, Andi Kleen wrote:
SNIP
Quoted 11 lines
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) calc_timer_values
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_cpu
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_rax
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) arch_perf_update_userpage
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __fentry__
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) using_native_sched_clock
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) sched_clock_stable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) perf_pmu_enable
perf 900 [000] 194167.205652203: ([kernel.kallsyms]) __x86_indirect_thunk_raxQuoted 5 lines
Also available in
git://git.kernel.org/pub/scm/linux/kernel/git/ak/linux-misc.gitpt/easy-script-2v1: Initial post
v2: Address review comments. Minor fixes to descriptions. Now builds everywhere.
with those changelogs updates:
thanks,
jirka