perf: add option to limit callchain stack scan to increase speed
v3 cover perf: add option to limit callchain stack scan to increase speed
v2->v3:
- Fix the commit log in the second patch which causes "git am" to fail.
- Remove the 1st patch in v2 as the problem will be fixed by another patch from Arnaldo.
v1->v2:
- Include a compilation fix patch and a code streamlining patch into the patch set.
- Use the __stringify() macro in stringify.h instead of adding a duplicate macro.
- Add the --max-stack option to perf-top as well.
This perf patch set contains the following changes:
Patch 1 - Streamline the append_chain() function to make it run a bit
faster.
Patch 2 - Add a --max-stack option to perf-report to speed up its
processing at the expense of less backtrace information available.
Patch 3 - Add a similar --max-stack option to perf-top.
Waiman Long (3):
perf: streamline append_chain() function
perf-report: add --max-stack option to limit callchain stack scan
perf-top: add --max-stack option to limit callchain stack scan
tools/perf/Documentation/perf-report.txt | 8 ++++++++ tools/perf/Documentation/perf-top.txt | 8 ++++++++ tools/perf/builtin-report.c | 22 +++++++++++++++++----- tools/perf/builtin-top.c | 9 +++++++-- tools/perf/util/callchain.c | 9 ++++----- tools/perf/util/machine.c | 14 +++++++++----- tools/perf/util/machine.h | 3 ++- tools/perf/util/session.c | 3 ++- tools/perf/util/top.h | 1 + 9 files changed, 58 insertions(+), 19 deletions(-)
PATCH v3 1/3 perf: streamline append_chain() function
When callgraph is enabled, the append_chain() function consumes a major portion of the total CPU time. This patch tries to streamline the append_chain() function by removing unneeded conditional test as well as using ?: statement which can be more efficient than the regular if statement in some architectures.
--- tools/perf/util/callchain.c | 9 ++++----- 1 files changed, 4 insertions(+), 5 deletions(-)
tools/perf/util/callchain.c +4 -5
@@ -315,6 +315,7 @@ append_chain(struct callchain_node *root,struct callchain_list *cnode;u64 start = cursor->pos;bool found = false; +bool func_mode = (callchain_param.key ==CCKEY_FUNCTION );u64 matches;/*
@@ -331,17 +332,15 @@ append_chain(struct callchain_node *root,if (!node)break ; - sym = node->sym; + sym = func_mode ? node->sym :NULL ; -if (cnode->ms.sym && sym && - callchain_param.key ==CCKEY_FUNCTION ) { +if (cnode->ms.sym && sym) {if (cnode->ms.sym->start != sym->start)break ; }else if (cnode->ip != node->ip)break ; -if (!found) - found = true; + found = true;callchain_cursor_advance (cursor); }
Signature · 1 line
1.7.1
PATCH v3 2/3 perf-report: add --max-stack option to limit callchain stack scan
In reply to Waiman LongWhen callgraph data was included in the perf data file, it may take a long time to scan all those data and merge them together especially if the stored callchains are long and the perf data file itself is large, like a Gbyte or so.
The callchain stack is currently limited to PERF_MAX_STACK_DEPTH (127). This is a large value. Usually the callgraph data that developers are most interested in are the first few levels, the rests are usually not looked at.
This patch adds a new --max-stack option to perf-report to limit the depth of callchain stack data to look at to reduce the time it takes for perf-report to finish its processing. It trades the presence of trailing stack information with faster speed.
The following table shows the elapsed time of doing perf-report on a perf.data file of size 985,531,828 bytes.
--max_stack Elapsed Time Output data size ----------- ------------ ---------------- not set 88.0s 124,422,651 64 87.5s 116,303,213 32 87.2s 112,023,804 16 86.6s 94,326,380 8 59.9s 33,697,248 4 40.7s 10,116,637 -g none 27.1s 2,555,810
--- tools/perf/Documentation/perf-report.txt | 8 ++++++++ tools/perf/builtin-report.c | 22 +++++++++++++++++----- tools/perf/builtin-top.c | 3 ++- tools/perf/util/machine.c | 14 +++++++++----- tools/perf/util/machine.h | 3 ++- tools/perf/util/session.c | 3 ++- 6 files changed, 40 insertions(+), 13 deletions(-)
tools/perf/Documentation/perf-report.txt +8 -0
@@ -135,6 +135,14 @@ OPTIONS Default: fractal,0.5,callee,function. +--max-stack:: + Set the stack depth limit when parsing the callchain, anything + beyond the specified depth will be ignored. This is a trade-off + between information loss and faster processing especially for + workloads that can have a very long callchain stack. + + Default: 127 + -G:: --inverted:: alias for inverted caller based call graph.
tools/perf/builtin-report.c +17 -5
@@ -47,6 +47,7 @@ struct perf_report {bool show_threads;bool inverted_callchain;bool mem_mode; +int max_stack;struct perf_read_values show_threads_values;const char *pretty_printing_style;const char *cpu_list;
@@ -88,7 +89,8 @@ static int perf_report__add_mem_hist_entry(struct perf_tool *tool,if ((sort__has_parent || symbol_conf.use_callchain) && sample->callchain) { err =machine__resolve_callchain (machine, evsel, al->thread, - sample, &parent, al); + sample, &parent, al, + rep->max_stack);if (err)return err; }
@@ -179,7 +181,8 @@ static int perf_report__add_branch_hist_entry(struct perf_tool *tool,if ((sort__has_parent || symbol_conf.use_callchain) && sample->callchain) { err =machine__resolve_callchain (machine, evsel, al->thread, - sample, &parent, al); + sample, &parent, al, + rep->max_stack);if (err)return err; }
@@ -242,18 +245,21 @@ out:return err; } -static int perf_evsel__add_hist_entry (struct perf_evsel *evsel, +static int perf_evsel__add_hist_entry (struct perf_tool *tool, +struct perf_evsel *evsel,struct addr_location *al,struct perf_sample *sample,struct machine *machine) { + struct perf_report *rep = container_of(tool, struct perf_report, tool);struct symbol *parent =NULL ;int err =0 ;struct hist_entry *he;if ((sort__has_parent || symbol_conf.use_callchain) && sample->callchain) { err =machine__resolve_callchain (machine, evsel, al->thread, - sample, &parent, al); + sample, &parent, al, + rep->max_stack);if (err)return err; }
@@ -330,7 +336,8 @@ static int process_sample_event(struct perf_tool *tool,if (al.map !=NULL ) al.map->dso->hit =1 ; - ret =perf_evsel__add_hist_entry (evsel, &al, sample, machine); + ret =perf_evsel__add_hist_entry (tool, evsel, &al, sample, + machine);if (ret <0 )pr_debug ("problem incrementing symbol period, skipping event\n" ); }
@@ -757,6 +764,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused) .ordered_samples = true, .ordering_requires_timestamps = true, }, + .max_stack = PERF_MAX_STACK_DEPTH, .pretty_printing_style = "normal", }; const struct option options[] = {
@@ -797,6 +805,10 @@ int cmd_report(int argc, const char **argv, const char *prefix __maybe_unused)OPT_CALLBACK_DEFAULT ('g' ,"call-graph" , &report,"output_type,min_percent[,print_limit],call_order" ,"Display callchains using output_type (graph, flat, fractal, or none) , min percent threshold, optional print limit, callchain order, key (function or address). " "Default: fractal,0.5,callee,function" , &parse_callchain_opt, callchain_default_opt), +OPT_INTEGER (0 ,"max-stack" , &report.max_stack, +"Set the maximum stack depth when parsing the callchain, " +"anything beyond the specified depth will be ignored. " +"Default: " __stringify(PERF_MAX_STACK_DEPTH )),OPT_BOOLEAN ('G' ,"inverted" , &report.inverted_callchain,"alias for inverted call graph" ), OPT_CALLBACK(0, "ignore-callees", NULL, "regex",
tools/perf/builtin-top.c +2 -1
@@ -771,7 +771,8 @@ static void perf_event__process_sample(struct perf_tool *tool, sample->callchain) { err =machine__resolve_callchain (machine, evsel, al.thread, sample, - &parent, &al); + &parent, &al, +PERF_MAX_STACK_DEPTH );if (err)return ; }
tools/perf/util/machine.c +9 -5
@@ -1267,10 +1267,12 @@ static int machine__resolve_callchain_sample(struct machine *machine,struct thread *thread, struct ip_callchain *chain, struct symbol **parent, - struct addr_location *root_al) + struct addr_location *root_al, + int max_stack) { u8 cpumode = PERF_RECORD_MISC_USER; -unsigned int i; +int chain_nr =min (max_stack, (int )chain->nr); +int i;int err;callchain_cursor_reset (&callchain_cursor);
@@ -1280,7 +1282,7 @@ static int machine__resolve_callchain_sample(struct machine *machine,return 0 ; } -for (i =0 ; i < chain->nr; i++) { +for (i =0 ; i < chain_nr; i++) {u64 ip;struct addr_location al;
@@ -1352,12 +1354,14 @@ int machine__resolve_callchain(struct machine *machine,struct thread *thread, struct perf_sample *sample, struct symbol **parent, - struct addr_location *root_al) + struct addr_location *root_al, + int max_stack) { int ret; ret =machine__resolve_callchain_sample (machine, thread, - sample->callchain, parent, root_al); + sample->callchain, parent, + root_al, max_stack);if (ret)return ret;
tools/perf/util/machine.h +2 -1
@@ -91,7 +91,8 @@ int machine__resolve_callchain(struct machine *machine,struct thread *thread, struct perf_sample *sample, struct symbol **parent, - struct addr_location *root_al); + struct addr_location *root_al, + int max_stack);/* * Default guest kernel is defined by parameter --guestkallsyms
tools/perf/util/session.c +2 -1
@@ -1525,7 +1525,8 @@ void perf_evsel__print_ip(struct perf_evsel *evsel, union perf_event *event,if (symbol_conf.use_callchain && sample->callchain) {if (machine__resolve_callchain (machine, evsel, al.thread, - sample,NULL ,NULL ) !=0 ) { + sample,NULL ,NULL , +PERF_MAX_STACK_DEPTH ) !=0 ) {if (verbose)error ("Failed to resolve callchain. Skipping\n" );return ;
Signature · 1 line
1.7.1
PATCH v3 3/3 perf-top: add --max-stack option to limit callchain stack scan
In reply to Waiman LongWhen the callgraph function is enabled (-G), it may take a long time to scan all the stack data and merge them accordingly.
This patch adds a new --max-stack option to perf-top to limit the depth of callchain stack data to look at to reduce the time it takes for perf-top to finish its processing. It reduces the amount of information provided to the user in exchange for faster speed.
--- tools/perf/Documentation/perf-top.txt | 8 ++++++++ tools/perf/builtin-top.c | 8 ++++++-- tools/perf/util/top.h | 1 + 3 files changed, 15 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-top.txt +8 -0
@@ -155,6 +155,14 @@ Default is to monitor all CPUS. Default: fractal,0.5,callee. +--max-stack:: + Set the stack depth limit when parsing the callchain, anything + beyond the specified depth will be ignored. This is a trade-off + between information loss and faster processing especially for + workloads that can have a very long callchain stack. + + Default: 127 + --ignore-callees=<regex>:: Ignore callees of the function(s) matching the given regex. This has the effect of collecting the callers of each such
tools/perf/builtin-top.c +6 -2
@@ -772,7 +772,7 @@ static void perf_event__process_sample(struct perf_tool *tool, err =machine__resolve_callchain (machine, evsel, al.thread, sample, &parent, &al, -PERF_MAX_STACK_DEPTH ); + top->max_stack);if (err)return ; }
@@ -1052,10 +1052,11 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused) .user_freq = UINT_MAX, .user_interval = ULLONG_MAX, .freq = 4000, /* 4 KHz */ - .target = { + .target = { .uses_mmap = true, }, }, + .max_stack = PERF_MAX_STACK_DEPTH, .sym_pcnt_filter = 5, };struct perf_record_opts *opts = &top.record_opts;
@@ -1110,6 +1111,9 @@ int cmd_top(int argc, const char **argv, const char *prefix __maybe_unused)OPT_CALLBACK_DEFAULT ('G' ,"call-graph" , &top.record_opts,"mode[,dump_size]" , record_callchain_help, &parse_callchain_opt,"fp" ), +OPT_INTEGER (0 ,"max-stack" , &top.max_stack, +"Set the maximum stack depth when parsing the callchain. " +"Default: " __stringify(PERF_MAX_STACK_DEPTH )),OPT_CALLBACK (0 ,"ignore-callees" ,NULL ,"regex" ,"ignore callees of these functions in call graphs" , report_parse_ignore_callees_opt),
tools/perf/util/top.h +1 -0
@@ -24,6 +24,7 @@ struct perf_top {u64 exact_samples;u64 guest_us_samples, guest_kernel_samples;int print_entries, count_filter, delay_secs; +int max_stack;bool hide_kernel_symbols, hide_user_symbols, zero;bool use_tui, use_stdio;bool kptr_restrict_warned;
Signature · 1 line
1.7.1
Namhyung Kim
maintainerHi Waiman,
Quoted 12 lines · Waiman Long
v2->v3:
- Fix the commit log in the second patch which causes "git am" to fail.
- Remove the 1st patch in v2 as the problem will be fixed by another patch from Arnaldo.
v1->v2:
- Include a compilation fix patch and a code streamlining patch into the patch set.
- Use the__stringify()macro instringify.hinstead of adding a duplicate macro.
- Add the --max-stack option to perf-top as well.
Quoted 8 lines
This perf patch set contains the following changes:
Patch 1 - Streamline the
append_chain()function to make it run a bit
faster.
Patch 2 - Add a --max-stack option to perf-report to speed up its
processing at the expense of less backtrace information available.
Patch 3 - Add a similar --max-stack option to perf-top.
At first, thank you for tackling this. It was really a problem using callchain on non trivial perf session. But there's another effort to address this problem in other direction. You might want to see below link [1]. Patch 1/8 does the thing and it seems Arnaldo already merged it into his tree.
Arnaldo, is there any issue prevents others from being merged?
[1] https://lkml.org/lkml/2013/10/11/16
Thanks,
Namhyung
Quoted 23 lines · Namhyung Kim
Hi Waiman,
Quoted 12 lines · Waiman Long
v2->v3:
- Fix the commit log in the second patch which causes "git am" to fail.
- Remove the 1st patch in v2 as the problem will be fixed by another patch from Arnaldo.
v1->v2:
- Include a compilation fix patch and a code streamlining patch into the patch set.
- Use the__stringify()macro instringify.hinstead of adding a duplicate macro.
- Add the --max-stack option to perf-top as well.Quoted 8 lines
This perf patch set contains the following changes:
Patch 1 - Streamline the
append_chain()function to make it run a bit
faster.
Patch 2 - Add a --max-stack option to perf-report to speed up its
processing at the expense of less backtrace information available.
Patch 3 - Add a similar --max-stack option to perf-top.
Quoted 5 lines
At first, thank you for tackling this. It was really a problem using callchain on non trivial perf session. But there's another effort to address this problem in other direction. You might want to see below link [1]. Patch 1/8 does the thing and it seems Arnaldo already merged it into his tree.
Thank for letting me aware of your patch to address the perf-report problem. As I am working on resolving scalability problem in large NUMA machines, I often encounters large perf.data files that took more than 1/2 hour for perf-report to produce anything. It is kind of frustrating to wait so long. I like the performance improvement in your patch 1/8, coupled with the max-stack option, it can really speed thing up.
-Longman