perf tool: Add sort key symoff for perf diff
PATCH v6 1/3 perf tool: Add sort key symoff for perf diff
From: Kan Liang <[email protected]>
Sometime, especially debugging scaling issue, the function level diff may be high granularity. The user may want to do deeper diff analysis for some cache or lock issue. The "symoff" key can let the user sort differential profile by ips.
Here is an example.
The source code for example_v1.c is as below:
noinline void f3(void)
{
volatile int i;
for (i = 0; i < 10000;) {
if(i%2)
i++;
else
i++;
}
}noinline void f2(void)
{
volatile int a = 100, b, c;
for (b = 0; b < 10000; b++)
c = a * b;}
noinline void f1(void)
{
f2();
f3();
}int main()
{
int i;
for (i = 0; i < 100000; i++)
f1();
}We run the example twice. (That's a simplified case. Of course, we cannot find the scaling issue here. )
[lk@localhost perf_diff]$ gcc example_v1.c -o example [lk@localhost perf_diff]$ perf record -o example_symoff_1.data ./example [ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.808 MB example_symoff_1.data (~35314 samples) ]
[lk@localhost perf_diff]$ perf record -o example_symoff_2.data ./example [ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.814 MB example_symoff_2.data (~35580 samples) ]
[lk@localhost perf_diff]$ perf diff -s symoff example_symoff_1.data example_symoff_2.data
Event 'cycles'
Baseline Delta Symbol + Offset
........ ....... ....................................
0.00% _raw_spin_lock+0x13
account_process_tick+0x15a
apic_timer_interrupt+0x0
check_preempt_wakeup+0x174
0.00% f1+0x0
0.00% f1+0x1
0.00% f1+0xe
0.00% f2+0xb
0.03% +0.02% f2+0x14
0.02% f2+0x1a
30.75% -0.18% f2+0x1d
7.66% -0.18% f2+0x20
0.00% f2+0x29
7.69% +0.12% f2+0x2c
0.01% f2+0x33
0.01% f2+0x34
0.00% f3+0x0
7.67% +0.15% f3+0xd
0.01% f3+0x10
f3+0x13
3.95% -0.21% f3+0x17
3.91% -0.05% f3+0x20
3.74% +0.04% f3+0x22
4.00% -0.10% f3+0x2b
30.44% +0.45% f3+0x2e
0.02% f3+0x35
0.02% f3+0x36
0.00% ktime_get+0x9c
0.00% main+0x1a
main+0x21
0.00% native_read_tsc+0x6
0.00% native_write_msr_safe+0xa
rcu_irq_enter+0x88
0.00% rcu_note_context_switch+0x12
0.00% run_timer_softirq+0x292
timekeeping_update.constprop.7+0x9c
0.00% timerqueue_add+0x50---
The patch is seperated from "[PATCH V4 0/3] perf tool: perf diff sort changes" patch set.
The first patch of the patch set has been merged.
The second patch to support perf diff different binaries was submitted by another thread.
This is the third patch to support symoff.
Changes since V4:
- Seperate from old patch set
- Symoff print format
- Check build_id
- Update man page
No change since V5
tools/perf/Documentation/perf-diff.txt | 2 +- tools/perf/Documentation/perf-report.txt | 1 + tools/perf/builtin-diff.c | 2 +- tools/perf/util/hist.c | 7 ++++ tools/perf/util/hist.h | 1 + tools/perf/util/sort.c | 67 ++++++++++++++++++++++++++++++++ tools/perf/util/sort.h | 2 + 7 files changed, 80 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-diff.txt +1 -1
@@ -50,7 +50,7 @@ OPTIONS -s:: --sort=:: - Sort by key(s): pid, comm, dso, symbol, cpu, parent, srcline. + Sort by key(s): pid, comm, dso, symbol, cpu, parent, srcline, symoff. Please see description of --sort in the perf-report man page. -t::
tools/perf/Documentation/perf-report.txt +1 -0
@@ -70,6 +70,7 @@ OPTIONS - pid: command and tid of the task - dso: name of library or module executed at the time of sample - symbol: name of function executed at the time of sample + - symoff: exact symbol + offset address executed at the time of sample. - parent: name of function matched to the parent regex filter. Unmatched entries are displayed as "[other]". - cpu: cpu number the task ran at the time of sample
tools/perf/builtin-diff.c +1 -1
@@ -744,7 +744,7 @@ static const struct option options[] = {OPT_STRING ('S' ,"symbols" , &symbol_conf.sym_list_str,"symbol[,symbol...]" ,"only consider these symbols" ),OPT_STRING ('s' ,"sort" , &sort_order,"key[,key2...]" , -"sort by key(s): pid, comm, dso, symbol, parent, cpu, srcline, ..." +"sort by key(s): pid, comm, dso, symbol, parent, cpu, srcline, symoff, ..." " Please refer the man page for the complete list." ),OPT_STRING ('t' ,"field-separator" , &symbol_conf.field_sep,"separator" , "separator for columns, no spaces will be added between "
tools/perf/util/hist.c +7 -0
@@ -59,6 +59,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)u16 len;/* +* +6 accounts for '"+0xYYY ' symoff info * +4 accounts for '[x] ' priv level info * +2 accounts for 0x prefix on raw addresses * +3 accounts for ' y ' symtab origin info
@@ -68,10 +69,16 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)if (verbose) symlen +=BITS_PER_LONG /4 +2 +3 ;hists__new_col_len (hists,HISTC_SYMBOL , symlen); + + symlen = h->ms.sym->namelen +6 ; +hists__new_col_len (hists,HISTC_SYMOFF , symlen); }else { symlen = unresolved_col_width +4 +2 ;hists__new_col_len (hists,HISTC_SYMBOL , symlen);hists__set_unres_dso_col_len (hists,HISTC_DSO ); + + symlen = unresolved_col_width +2 ; +hists__new_col_len (hists,HISTC_SYMOFF , symlen); } len =thread__comm_len (h->thread);
tools/perf/util/hist.h +1 -0
@@ -24,6 +24,7 @@ enum hist_filter {enum hist_column {HISTC_SYMBOL , +HISTC_SYMOFF ,HISTC_DSO ,HISTC_THREAD ,HISTC_COMM ,
tools/perf/util/sort.c +67 -0
85 lines not on this page · raw message
tools/perf/util/sort.h +2 -0
@@ -38,6 +38,7 @@ extern enum sort_mode sort__mode;extern struct sort_entry sort_comm;extern struct sort_entry sort_dso;extern struct sort_entry sort_sym; +extern struct sort_entry sort_symoff;extern struct sort_entry sort_parent;extern struct sort_entry sort_dso_from;extern struct sort_entry sort_dso_to;
@@ -161,6 +162,7 @@ enum sort_type {SORT_COMM ,SORT_DSO ,SORT_SYM , +SORT_SYMOFF ,SORT_PARENT ,SORT_CPU , SORT_SRCLINE,
Signature · 1 line
1.8.3.2
PATCH v6 2/3 perf tool: new function to compare common part of build-ids
From: Kan Liang <[email protected]>
New function to compare the build_ids between different DSOs This function is only interested in the common part of build_ids. For example, dsos_a includes DSO A and DSO B. dsos_b includes DSO B and DSO C. Only DSO B's build_id will be compared.
---
Changes since V5:
- Replace Arnaldo's dsos__build_ids_equal by bool dsos__build_ids_common_equal
- Only compare the common part of build-ids
tools/perf/util/dso.c | 20 ++++++++++++++++++++ tools/perf/util/dso.h | 2 ++ 2 files changed, 22 insertions(+)
tools/perf/util/dso.c +20 -0
@@ -1087,3 +1087,23 @@ enum dso_type dso__type(struct dso *dso, struct machine *machine)return dso__type_fd (fd); } + +bool dsos__build_ids_common_equal (struct dsos *dsos_a,struct dsos *dsos_b) +{ +struct dso *dso_a, *dso_b; + +list_for_each_entry (dso_a, &dsos_a->head, node) { + dso_b =dsos__find (dsos_b, dso_a->short_name, true); +if (dso_b ==NULL ) +continue ; +/* +* Only compare the common part of dsos. +*/ +if (dso_a->has_build_id && dso_b->has_build_id) { +if (memcmp (dso_a->build_id, dso_b->build_id,sizeof (dso_a->build_id))) +return false; + } + } + +return true; +}
tools/perf/util/dso.h +2 -0
@@ -278,4 +278,6 @@ void dso__free_a2l(struct dso *dso);enum dso_type dso__type (struct dso *dso,struct machine *machine); +bool dsos__build_ids_common_equal (struct dsos *dsos_a,struct dsos *dsos_b); +#endif /* __PERF_DSO */
Signature · 1 line
1.8.3.2
PATCH v6 3/3 perf tool: check buildid for symoff
In reply to Kan LiangFrom: Kan Liang <[email protected]>
symoff can support both same binaries and different binaries. However, the offset may be changed for different binaries. This patch checks the buildid of perf.data. If they are from different binaries, print a warning to notify the user.
---
Changes since V5:
- Warning once.
- Print build-ids with --verbose
tools/perf/builtin-diff.c | 41 +++++++++++++++++++++++++++++++++++++++++ tools/perf/util/sort.c | 3 +++ tools/perf/util/sort.h | 1 + 3 files changed, 45 insertions(+)
tools/perf/builtin-diff.c +41 -0
@@ -17,6 +17,7 @@#include "util/symbol.h" #include "util/util.h" #include "util/data.h" +#include "asm/bug.h" #include <stdlib.h> #include <math.h>
@@ -678,6 +679,43 @@ static void data__free(struct data__file *d) } } +static void buildid_check (void ) +{ +struct dsos *base_k_dsos = &data__files[0 ].session->machines.host.kernel_dsos; +struct dsos *base_u_dsos = &data__files[0 ].session->machines.host.user_dsos; +struct dsos *k_dsos_tmp, *u_dsos_tmp; +struct data__file *d; +bool k_warn, u_warn; +bool first = true; +int i; + +data__for_each_file_new (i, d) { + k_dsos_tmp = &d->session->machines.host.kernel_dsos; + u_dsos_tmp = &d->session->machines.host.user_dsos; + + k_warn = !dsos__build_ids_common_equal (base_k_dsos, k_dsos_tmp); + u_warn = !dsos__build_ids_common_equal (base_u_dsos, u_dsos_tmp); + +WARN_ONCE (k_warn | u_warn,"The perf.data come from different %s%s%s. " +"The symbol offset may vary. Please apply --verbose for details.\n" , + k_warn ?"kernel" :"" , + (k_warn && u_warn) ?" and " :"" , + u_warn ?"user binary" :"" ); + +if (verbose) { +if (first) { +fprintf (stdout,"# ========%s (Baseline) build-ids\n" , data__files[0 ].file.path); +machine__fprintf_dsos_buildid (&data__files[0 ].session->machines.host, stdout,NULL ,0 ); +fprintf (stdout,"# ========\n#\n" ); + first = false; + } +fprintf (stdout,"# ========%s build-ids\n" , d->file.path); +machine__fprintf_dsos_buildid (&d->session->machines.host, stdout,NULL ,0 ); +fprintf (stdout,"# ========\n#\n" ); + } + } +} +static int __cmd_diff (void ) {struct data__file *d;
@@ -700,6 +738,9 @@ static int __cmd_diff(void)perf_evlist__collapse_resort (d->session->evlist); } +if (sort__has_symoff) +buildid_check (); +data_process (); out_delete:
tools/perf/util/sort.c +3 -0
@@ -21,6 +21,7 @@ int sort__need_collapse = 0;int sort__has_parent =0 ;int sort__has_sym =0 ;int sort__has_dso =0 ; +int sort__has_symoff =0 ;enum sort_mode sort__mode =SORT_MODE__NORMAL ;
@@ -1495,6 +1496,7 @@ int sort_dimension__add(const char *tok) }else if (sd->entry == &sort_dso) { sort__has_dso =1 ; }else if (sd->entry == &sort_symoff) { + sort__has_symoff =1 ;/* * For symoff sort key, not only the offset but also the * symbol name should be compared.
@@ -1879,6 +1881,7 @@ void reset_output_field(void) sort__has_parent =0 ; sort__has_sym =0 ; sort__has_dso =0 ; + sort__has_symoff =0 ; field_order =NULL ; sort_order =NULL ;
tools/perf/util/sort.h +1 -0
@@ -34,6 +34,7 @@ extern int have_ignore_callees;extern int sort__need_collapse;extern int sort__has_parent;extern int sort__has_sym; +extern int sort__has_symoff;extern enum sort_mode sort__mode;extern struct sort_entry sort_comm;extern struct sort_entry sort_dso;
Signature · 1 line
1.8.3.2
On Mon, Dec 01, 2014 at 09:40:10AM -0500, Kan Liang wrote:
SNIP
Quoted 42 lines
tools/perf/util/sort.c +42
-0+static int64_t +sort__symoff_collapse (struct hist_entry *left,struct hist_entry *right) +{ +struct symbol *sym_l = left->ms.sym; +struct symbol *sym_r = right->ms.sym; +u64 symoff_l, symoff_r; +int64_t ret; + +if (!sym_l || !sym_r) +return cmp_null (sym_l, sym_r); + + ret =strcmp (sym_r->name, sym_l->name); +if (ret) +return ret; + + + symoff_l = left->ip - sym_l->start; + symoff_r = right->ip - sym_r->start; + +return (int64_t )(symoff_r - symoff_l); +} + +static int hist_entry__symoff_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) +{ +struct map *map = he->ms.map; +struct symbol *sym = he->ms.sym; +size_t ret =0 ; + +if (sym) { + ret +=repsep_snprintf (bf + ret, size - ret,"%s" , sym->name); + ret +=repsep_snprintf (bf + ret, size - ret,"+0x%llx" , + he->ip - sym->start); + + }else { +size_t len =BITS_PER_LONG /4 ; + + ret +=repsep_snprintf (bf + ret, size - ret,"%-#.*llx" , len, + map ? map->unmap_ip (map, he->ip) : he->ip); + } + + ret += repsep_snprintf(bf + ret, size - ret, "%-*s",
tools/perf/util/sort.c struct sort_entry sort_sym = { · line 340
Quoted 10 lines
tools/perf/util/sort.c +10
-0+ width - ret, ""); +return ret; +} + +struct sort_entry sort_symoff = { + .se_header ="Symbol + Offset" , + .se_cmp = sort__symoff_cmp, + .se_snprintf = hist_entry__symoff_snprintf, + .se_width_idx =HISTC_SYMOFF , +};
I might have missed this in previous discussions,
but do we also want just pure string comparison?
now I get:
0.30% +0.42% main+0x110
1.80% -0.45% main+0x115
0.05% -0.04% main+0x118
0.34% +0.49% main+0x11c
2.15% -0.22% main+0x120
0.41% +0.22% main+0x123
1.86% +0.04% main+0x12f
3.86% -0.69% main+0x133
0.02% main+0x137
3.80% -1.19% main+0x13d
0.26% +0.45% main+0x141
2.26% +1.41% main+0x145
8.78% -1.59% main+0x148
0.05% main+0x14c
1.40% -0.26% main+0x155
0.07% main+0x158
0.74% -0.03% main+0x15b
1.06% -0.17% main+0x160
0.31% +0.30% main+0x1a8
1.82% -0.51% main+0x1af
0.09% +0.07% main+0x1b1
0.05% main+0x1b4 could we add something like '-s symstr' to do
only symbol string comparison, so the previous
output would gather in single line like:
5.09% +2.07% main
otherwise the patchset looks ok to me:
thanks,
jirka
Quoted 86 lines · Kan Liang
SNIP
Quoted 30 lines
tools/perf/util/sort.c +29
-0+static int64_t +sort__symoff_collapse (struct hist_entry *left,struct hist_entry +*right) { +struct symbol *sym_l = left->ms.sym; +struct symbol *sym_r = right->ms.sym; +u64 symoff_l, symoff_r; +int64_t ret; + +if (!sym_l || !sym_r) +return cmp_null (sym_l, sym_r); + + ret =strcmp (sym_r->name, sym_l->name); +if (ret) +return ret; + + + symoff_l = left->ip - sym_l->start; + symoff_r = right->ip - sym_r->start; + +return (int64_t )(symoff_r - symoff_l); } + +static int hist_entry__symoff_snprintf (struct hist_entry *he,char *bf, +size_t size,unsigned int width) { +struct map *map = he->ms.map; +struct symbol *sym = he->ms.sym; +size_t ret =0 ; + +if (sym) { + ret += repsep_snprintf(bf + ret, size - ret, "%s", sym-name);
tools/perf/util/sort.c struct sort_entry sort_sym = {
Quoted 8 lines
tools/perf/util/sort.c +8
-0+ ret +=repsep_snprintf (bf + ret, size - ret,"+0x%llx" , + he->ip - sym->start); + + }else { +size_t len =BITS_PER_LONG /4 ; + + ret += repsep_snprintf(bf + ret, size - ret, "%-#.*llx", len, + map ? map->unmap_ip(map, he->ip) :
he->ip);tools/perf/util/sort.c struct sort_entry sort_sym = { · line 340
Quoted 13 lines
tools/perf/util/sort.c +13
-0+ } + + ret +=repsep_snprintf (bf + ret, size - ret,"%-*s" , + width - ret,"" ); +return ret; +} + +struct sort_entry sort_symoff = { + .se_header ="Symbol + Offset" , + .se_cmp = sort__symoff_cmp, + .se_snprintf = hist_entry__symoff_snprintf, + .se_width_idx =HISTC_SYMOFF , +};I might have missed this in previous discussions, but do we also want just pure string comparison?
now I get:
0.30% +0.42% main+0x110 1.80% -0.45% main+0x115 0.05% -0.04% main+0x118 0.34% +0.49% main+0x11c 2.15% -0.22% main+0x120 0.41% +0.22% main+0x123 1.86% +0.04% main+0x12f 3.86% -0.69% main+0x133 0.02% main+0x137 3.80% -1.19% main+0x13d 0.26% +0.45% main+0x141 2.26% +1.41% main+0x145 8.78% -1.59% main+0x148 0.05% main+0x14c 1.40% -0.26% main+0x155 0.07% main+0x158 0.74% -0.03% main+0x15b 1.06% -0.17% main+0x160 0.31% +0.30% main+0x1a8 1.82% -0.51% main+0x1af 0.09% +0.07% main+0x1b1 0.05% main+0x1b4
Quoted 4 lines
could we add something like '-s symstr' to do only symbol string comparison, so the previous output would gather in single line like:
5.09% +2.07% main
Yes, we have a patch to do that.
https://lkml.org/lkml/2014/11/21/367
The default sort key "symbol" do symbols names comparison.
The symoff here is an extension in case anyone want to do deeper analysis.
Thanks,
Kan
Quoted 8 lines
otherwise the patchset looks ok to me:
Acked-by Jiri Olsa <[email protected]>thanks,
jirka
On Mon, Dec 01, 2014 at 08:05:48PM +0000, Liang, Kan wrote:
SNIP
Quoted 13 lines
Quoted 13 lines
0.05% main+0x14c 1.40% -0.26% main+0x155 0.07% main+0x158 0.74% -0.03% main+0x15b 1.06% -0.17% main+0x160 0.31% +0.30% main+0x1a8 1.82% -0.51% main+0x1af 0.09% +0.07% main+0x1b1 0.05% main+0x1b4could we add something like '-s symstr' to do only symbol string comparison, so the previous output would gather in single line like:
Quoted 7 lines
Quoted 1 line
5.09% +2.07% mainYes, we have a patch to do that.
https://lkml.org/lkml/2014/11/21/367
The default sort key "symbol" do symbols names comparison.The symoff here is an extension in case anyone want to do deeper analysis.
cool, I just acked it
thanks,
jirka