Skip to contentSkip to conversation

perf tool: Add sort key symoff for perf diff

PATCH In review Kan Liang 2 participants 6 messages Dec 1, 2014 perf events

lore
Share & tools
lore

Kan Liang

author
#0 lore raw

PATCH v6 1/3 perf tool: Add sort key symoff for perf diff

From: Kan Liang <[email protected]>

Sometime, especially debugging scaling issue, the function level diff may be high granularity. The user may want to do deeper diff analysis for some cache or lock issue. The "symoff" key can let the user sort differential profile by ips.

Here is an example.

The source code for example_v1.c is as below:

noinline void f3(void)

{
        volatile int i;
        for (i = 0; i < 10000;) {
                if(i%2)
                        i++;
                else
                        i++;
        }
}

noinline void f2(void)

{
        volatile int a = 100, b, c;
        for (b = 0; b < 10000; b++)
                c = a * b;

}

noinline void f1(void)

{
                f2();
                f3();
}

int main()

{
        int i;
        for (i = 0; i < 100000; i++)
                f1();
}

We run the example twice. (That's a simplified case. Of course, we cannot find the scaling issue here. )

[lk@localhost perf_diff]$ gcc example_v1.c -o example [lk@localhost perf_diff]$ perf record -o example_symoff_1.data ./example [ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.808 MB example_symoff_1.data (~35314 samples) ]

[lk@localhost perf_diff]$ perf record -o example_symoff_2.data ./example [ perf record: Woken up 4 times to write data ]
[ perf record: Captured and wrote 0.814 MB example_symoff_2.data (~35580 samples) ]

[lk@localhost perf_diff]$ perf diff -s symoff example_symoff_1.data example_symoff_2.data

Event 'cycles'

Baseline Delta Symbol + Offset
........ ....... ....................................

     0.00%           _raw_spin_lock+0x13
                     account_process_tick+0x15a
                     apic_timer_interrupt+0x0
                     check_preempt_wakeup+0x174
     0.00%           f1+0x0
     0.00%           f1+0x1
     0.00%           f1+0xe
     0.00%           f2+0xb
     0.03%   +0.02%  f2+0x14
     0.02%           f2+0x1a
    30.75%   -0.18%  f2+0x1d
     7.66%   -0.18%  f2+0x20
     0.00%           f2+0x29
     7.69%   +0.12%  f2+0x2c
     0.01%           f2+0x33
     0.01%           f2+0x34
     0.00%           f3+0x0
     7.67%   +0.15%  f3+0xd
     0.01%           f3+0x10
                     f3+0x13
     3.95%   -0.21%  f3+0x17
     3.91%   -0.05%  f3+0x20
     3.74%   +0.04%  f3+0x22
     4.00%   -0.10%  f3+0x2b
    30.44%   +0.45%  f3+0x2e
     0.02%           f3+0x35
     0.02%           f3+0x36
     0.00%           ktime_get+0x9c
     0.00%           main+0x1a
                     main+0x21
     0.00%           native_read_tsc+0x6
     0.00%           native_write_msr_safe+0xa
                     rcu_irq_enter+0x88
     0.00%           rcu_note_context_switch+0x12
     0.00%           run_timer_softirq+0x292
                     timekeeping_update.constprop.7+0x9c
     0.00%           timerqueue_add+0x50

---

The patch is seperated from "[PATCH V4 0/3] perf tool: perf diff sort changes" patch set.
The first patch of the patch set has been merged.
The second patch to support perf diff different binaries was submitted by another thread.
This is the third patch to support symoff.

Changes since V4:
- Seperate from old patch set
- Symoff print format
- Check build_id
- Update man page

No change since V5

 tools/perf/Documentation/perf-diff.txt   |  2 +-
 tools/perf/Documentation/perf-report.txt |  1 +
 tools/perf/builtin-diff.c                |  2 +-
 tools/perf/util/hist.c                   |  7 ++++
 tools/perf/util/hist.h                   |  1 +
 tools/perf/util/sort.c                   | 67 ++++++++++++++++++++++++++++++++
 tools/perf/util/sort.h                   |  2 +
 7 files changed, 80 insertions(+), 2 deletions(-)
tools/perf/Documentation/perf-diff.txt +1 -1
index e463caa..4179ddfa 100644
@@ -50,7 +50,7 @@ OPTIONS
 
 -s::
 --sort=::
-	Sort by key(s): pid, comm, dso, symbol, cpu, parent, srcline.
+	Sort by key(s): pid, comm, dso, symbol, cpu, parent, srcline, symoff.
 	Please see description of --sort in the perf-report man page.
 
 -t::
tools/perf/Documentation/perf-report.txt +1 -0
index 0927bf4..5b63311 100644
@@ -70,6 +70,7 @@ OPTIONS
 	- pid: command and tid of the task
 	- dso: name of library or module executed at the time of sample
 	- symbol: name of function executed at the time of sample
+	- symoff: exact symbol + offset address executed at the time of sample.
 	- parent: name of function matched to the parent regex filter. Unmatched
 	entries are displayed as "[other]".
 	- cpu: cpu number the task ran at the time of sample
tools/perf/builtin-diff.c +1 -1
index 1ce425d..03a4001 100644
@@ -744,7 +744,7 @@ static const struct option options[] = {
 	OPT_STRING('S', "symbols", &symbol_conf.sym_list_str, "symbol[,symbol...]",
 		   "only consider these symbols"),
 	OPT_STRING('s', "sort", &sort_order, "key[,key2...]",
-		   "sort by key(s): pid, comm, dso, symbol, parent, cpu, srcline, ..."
+		   "sort by key(s): pid, comm, dso, symbol, parent, cpu, srcline, symoff, ..."
 		   " Please refer the man page for the complete list."),
 	OPT_STRING('t', "field-separator", &symbol_conf.field_sep, "separator",
 		   "separator for columns, no spaces will be added between "
tools/perf/util/hist.c +7 -0
index 6e88b9e..062c3d4b 100644
@@ -59,6 +59,7 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 	u16 len;
 
 	/*
+	 * +6 accounts for '"+0xYYY ' symoff info
 	 * +4 accounts for '[x] ' priv level info
 	 * +2 accounts for 0x prefix on raw addresses
 	 * +3 accounts for ' y ' symtab origin info
@@ -68,10 +69,16 @@ void hists__calc_col_len(struct hists *hists, struct hist_entry *h)
 		if (verbose)
 			symlen += BITS_PER_LONG / 4 + 2 + 3;
 		hists__new_col_len(hists, HISTC_SYMBOL, symlen);
+
+		symlen = h->ms.sym->namelen + 6;
+		hists__new_col_len(hists, HISTC_SYMOFF, symlen);
 	} else {
 		symlen = unresolved_col_width + 4 + 2;
 		hists__new_col_len(hists, HISTC_SYMBOL, symlen);
 		hists__set_unres_dso_col_len(hists, HISTC_DSO);
+
+		symlen = unresolved_col_width + 2;
+		hists__new_col_len(hists, HISTC_SYMOFF, symlen);
 	}
 
 	len = thread__comm_len(h->thread);
tools/perf/util/hist.h +1 -0
index d0ef9a1..874b203 100644
@@ -24,6 +24,7 @@ enum hist_filter {
 
 enum hist_column {
 	HISTC_SYMBOL,
+	HISTC_SYMOFF,
 	HISTC_DSO,
 	HISTC_THREAD,
 	HISTC_COMM,
tools/perf/util/sort.c +67 -0

85 lines not on this page · raw message

tools/perf/util/sort.h +2 -0
index c03e4ff..ea0122f 100644
@@ -38,6 +38,7 @@ extern enum sort_mode sort__mode;
 extern struct sort_entry sort_comm;
 extern struct sort_entry sort_dso;
 extern struct sort_entry sort_sym;
+extern struct sort_entry sort_symoff;
 extern struct sort_entry sort_parent;
 extern struct sort_entry sort_dso_from;
 extern struct sort_entry sort_dso_to;
@@ -161,6 +162,7 @@ enum sort_type {
 	SORT_COMM,
 	SORT_DSO,
 	SORT_SYM,
+	SORT_SYMOFF,
 	SORT_PARENT,
 	SORT_CPU,
 	SORT_SRCLINE,
Signature · 1 line
1.8.3.2
author
#1 lore raw

PATCH v6 2/3 perf tool: new function to compare common part of build-ids

From: Kan Liang <[email protected]>

New function to compare the build_ids between different DSOs This function is only interested in the common part of build_ids. For example, dsos_a includes DSO A and DSO B. dsos_b includes DSO B and DSO C. Only DSO B's build_id will be compared.

---

Changes since V5:
- Replace Arnaldo's dsos__build_ids_equal by bool dsos__build_ids_common_equal
- Only compare the common part of build-ids

 tools/perf/util/dso.c | 20 ++++++++++++++++++++
 tools/perf/util/dso.h |  2 ++
 2 files changed, 22 insertions(+)
tools/perf/util/dso.c +20 -0
index 45be944..04416b9 100644
@@ -1087,3 +1087,23 @@ enum dso_type dso__type(struct dso *dso, struct machine *machine)
 
 	return dso__type_fd(fd);
 }
+
+bool dsos__build_ids_common_equal(struct dsos *dsos_a, struct dsos *dsos_b)
+{
+	struct dso *dso_a, *dso_b;
+
+	list_for_each_entry(dso_a, &dsos_a->head, node) {
+		dso_b = dsos__find(dsos_b, dso_a->short_name, true);
+		if (dso_b == NULL)
+			continue;
+		/*
+		 * Only compare the common part of dsos.
+		 */
+		if (dso_a->has_build_id && dso_b->has_build_id) {
+			if (memcmp(dso_a->build_id, dso_b->build_id, sizeof(dso_a->build_id)))
+				return false;
+		}
+	}
+
+	return true;
+}
tools/perf/util/dso.h +2 -0
index 3782c82..e92217c 100644
@@ -278,4 +278,6 @@ void dso__free_a2l(struct dso *dso);
 
 enum dso_type dso__type(struct dso *dso, struct machine *machine);
 
+bool dsos__build_ids_common_equal(struct dsos *dsos_a, struct dsos *dsos_b);
+
 #endif /* __PERF_DSO */
Signature · 1 line
1.8.3.2
author
#2 lore raw

PATCH v6 3/3 perf tool: check buildid for symoff

In reply to Kan Liang

From: Kan Liang <[email protected]>

symoff can support both same binaries and different binaries. However, the offset may be changed for different binaries. This patch checks the buildid of perf.data. If they are from different binaries, print a warning to notify the user.

---

Changes since V5:
- Warning once.
- Print build-ids with --verbose

 tools/perf/builtin-diff.c | 41 +++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/sort.c    |  3 +++
 tools/perf/util/sort.h    |  1 +
 3 files changed, 45 insertions(+)
tools/perf/builtin-diff.c +41 -0
index 03a4001..7f1cb6a 100644
@@ -17,6 +17,7 @@
 #include "util/symbol.h"
 #include "util/util.h"
 #include "util/data.h"
+#include "asm/bug.h"
 
 #include <stdlib.h>
 #include <math.h>
@@ -678,6 +679,43 @@ static void data__free(struct data__file *d)
 	}
 }
 
+static void buildid_check(void)
+{
+	struct dsos *base_k_dsos = &data__files[0].session->machines.host.kernel_dsos;
+	struct dsos *base_u_dsos = &data__files[0].session->machines.host.user_dsos;
+	struct dsos *k_dsos_tmp, *u_dsos_tmp;
+	struct data__file *d;
+	bool k_warn, u_warn;
+	bool first = true;
+	int i;
+
+	data__for_each_file_new(i, d) {
+		k_dsos_tmp = &d->session->machines.host.kernel_dsos;
+		u_dsos_tmp = &d->session->machines.host.user_dsos;
+
+		k_warn = !dsos__build_ids_common_equal(base_k_dsos, k_dsos_tmp);
+		u_warn = !dsos__build_ids_common_equal(base_u_dsos, u_dsos_tmp);
+
+		WARN_ONCE(k_warn | u_warn, "The perf.data come from different %s%s%s. "
+					   "The symbol offset may vary. Please apply --verbose for details.\n",
+					   k_warn ? "kernel" : "",
+					   (k_warn && u_warn) ? " and " : "",
+					   u_warn ? "user binary" : "");
+
+		if (verbose) {
+			if (first) {
+				fprintf(stdout, "# ========%s (Baseline) build-ids\n", data__files[0].file.path);
+				machine__fprintf_dsos_buildid(&data__files[0].session->machines.host, stdout, NULL, 0);
+				fprintf(stdout, "# ========\n#\n");
+				first = false;
+			}
+			fprintf(stdout, "# ========%s build-ids\n", d->file.path);
+			machine__fprintf_dsos_buildid(&d->session->machines.host, stdout, NULL, 0);
+			fprintf(stdout, "# ========\n#\n");
+		}
+	}
+}
+
 static int __cmd_diff(void)
 {
 	struct data__file *d;
@@ -700,6 +738,9 @@ static int __cmd_diff(void)
 		perf_evlist__collapse_resort(d->session->evlist);
 	}
 
+	if (sort__has_symoff)
+		buildid_check();
+
 	data_process();
 
  out_delete:
tools/perf/util/sort.c +3 -0
index 2a1df9f..07cfc4f 100644
@@ -21,6 +21,7 @@ int		sort__need_collapse = 0;
 int		sort__has_parent = 0;
 int		sort__has_sym = 0;
 int		sort__has_dso = 0;
+int		sort__has_symoff = 0;
 enum sort_mode	sort__mode = SORT_MODE__NORMAL;
 
 
@@ -1495,6 +1496,7 @@ int sort_dimension__add(const char *tok)
 		} else if (sd->entry == &sort_dso) {
 			sort__has_dso = 1;
 		} else if (sd->entry == &sort_symoff) {
+			sort__has_symoff = 1;
 			/*
 			 * For symoff sort key, not only the offset but also the
 			 * symbol name should be compared.
@@ -1879,6 +1881,7 @@ void reset_output_field(void)
 	sort__has_parent = 0;
 	sort__has_sym = 0;
 	sort__has_dso = 0;
+	sort__has_symoff = 0;
 
 	field_order = NULL;
 	sort_order = NULL;
tools/perf/util/sort.h +1 -0
index ea0122f..d2f9782 100644
@@ -34,6 +34,7 @@ extern int have_ignore_callees;
 extern int sort__need_collapse;
 extern int sort__has_parent;
 extern int sort__has_sym;
+extern int sort__has_symoff;
 extern enum sort_mode sort__mode;
 extern struct sort_entry sort_comm;
 extern struct sort_entry sort_dso;
Signature · 1 line
1.8.3.2
Acked-by on v6 1/3
#3 lore raw
In reply to Kan Liang

On Mon, Dec 01, 2014 at 09:40:10AM -0500, Kan Liang wrote:

SNIP

Quoted 42 lines
tools/perf/util/sort.c +42 -0
+static int64_t
+sort__symoff_collapse(struct hist_entry *left, struct hist_entry *right)
+{
+	struct symbol *sym_l = left->ms.sym;
+	struct symbol *sym_r = right->ms.sym;
+	u64 symoff_l, symoff_r;
+	int64_t ret;
+
+	if (!sym_l || !sym_r)
+		return cmp_null(sym_l, sym_r);
+
+	ret = strcmp(sym_r->name, sym_l->name);
+	if (ret)
+		return ret;
+
+
+	symoff_l = left->ip - sym_l->start;
+	symoff_r = right->ip - sym_r->start;
+
+	return (int64_t)(symoff_r - symoff_l);
+}
+
+static int hist_entry__symoff_snprintf(struct hist_entry *he, char *bf,
+				    size_t size, unsigned int width)
+{
+	struct map *map = he->ms.map;
+	struct symbol *sym = he->ms.sym;
+	size_t ret = 0;
+
+	if (sym) {
+		ret += repsep_snprintf(bf + ret, size - ret, "%s", sym->name);
+		ret += repsep_snprintf(bf + ret, size - ret, "+0x%llx",
+				       he->ip - sym->start);
+
+	} else {
+		size_t len = BITS_PER_LONG / 4;
+
+		ret += repsep_snprintf(bf + ret, size - ret, "%-#.*llx", len,
+				       map ? map->unmap_ip(map, he->ip) : he->ip);
+	}
+
+	ret += repsep_snprintf(bf + ret, size - ret, "%-*s",

tools/perf/util/sort.c struct sort_entry sort_sym = { · line 340

Quoted 10 lines
tools/perf/util/sort.c +10 -0
+			       width - ret, "");
+	return ret;
+}
+
+struct sort_entry sort_symoff = {
+	.se_header	= "Symbol + Offset",
+	.se_cmp		= sort__symoff_cmp,
+	.se_snprintf	= hist_entry__symoff_snprintf,
+	.se_width_idx	= HISTC_SYMOFF,
+};

I might have missed this in previous discussions,
but do we also want just pure string comparison?

now I get:

     0.30%   +0.42%  main+0x110                         
     1.80%   -0.45%  main+0x115                         
     0.05%   -0.04%  main+0x118                         
     0.34%   +0.49%  main+0x11c                         
     2.15%   -0.22%  main+0x120                         
     0.41%   +0.22%  main+0x123                         
     1.86%   +0.04%  main+0x12f                         
     3.86%   -0.69%  main+0x133                         
     0.02%           main+0x137                         
     3.80%   -1.19%  main+0x13d                         
     0.26%   +0.45%  main+0x141                         
     2.26%   +1.41%  main+0x145                         
     8.78%   -1.59%  main+0x148                         
     0.05%           main+0x14c                         
     1.40%   -0.26%  main+0x155                         
     0.07%           main+0x158                         
     0.74%   -0.03%  main+0x15b                         
     1.06%   -0.17%  main+0x160                         
     0.31%   +0.30%  main+0x1a8                         
     1.82%   -0.51%  main+0x1af                         
     0.09%   +0.07%  main+0x1b1                         
     0.05%           main+0x1b4                         

could we add something like '-s symstr' to do
only symbol string comparison, so the previous
output would gather in single line like:

     5.09%   +2.07%  main

otherwise the patchset looks ok to me:

thanks,
jirka

author
#4 lore raw
Quoted 86 lines · Kan Liang

SNIP

Quoted 30 lines
tools/perf/util/sort.c +29 -0
+static int64_t
+sort__symoff_collapse(struct hist_entry *left, struct hist_entry
+*right) {
+	struct symbol *sym_l = left->ms.sym;
+	struct symbol *sym_r = right->ms.sym;
+	u64 symoff_l, symoff_r;
+	int64_t ret;
+
+	if (!sym_l || !sym_r)
+		return cmp_null(sym_l, sym_r);
+
+	ret = strcmp(sym_r->name, sym_l->name);
+	if (ret)
+		return ret;
+
+
+	symoff_l = left->ip - sym_l->start;
+	symoff_r = right->ip - sym_r->start;
+
+	return (int64_t)(symoff_r - symoff_l); }
+
+static int hist_entry__symoff_snprintf(struct hist_entry *he, char *bf,
+				    size_t size, unsigned int width) {
+	struct map *map = he->ms.map;
+	struct symbol *sym = he->ms.sym;
+	size_t ret = 0;
+
+	if (sym) {
+		ret += repsep_snprintf(bf + ret, size - ret, "%s", sym-

name);

tools/perf/util/sort.c struct sort_entry sort_sym = {

Quoted 8 lines
tools/perf/util/sort.c +8 -0
+		ret += repsep_snprintf(bf + ret, size - ret, "+0x%llx",
+				       he->ip - sym->start);
+
+	} else {
+		size_t len = BITS_PER_LONG / 4;
+
+		ret += repsep_snprintf(bf + ret, size - ret, "%-#.*llx", len,
+				       map ? map->unmap_ip(map, he->ip) :

he->ip);

tools/perf/util/sort.c struct sort_entry sort_sym = { · line 340

Quoted 13 lines
tools/perf/util/sort.c +13 -0
+	}
+
+	ret += repsep_snprintf(bf + ret, size - ret, "%-*s",
+			       width - ret, "");
+	return ret;
+}
+
+struct sort_entry sort_symoff = {
+	.se_header	= "Symbol + Offset",
+	.se_cmp		= sort__symoff_cmp,
+	.se_snprintf	= hist_entry__symoff_snprintf,
+	.se_width_idx	= HISTC_SYMOFF,
+};

I might have missed this in previous discussions, but do we also want just pure string comparison?

now I get:

     0.30%   +0.42%  main+0x110
     1.80%   -0.45%  main+0x115
     0.05%   -0.04%  main+0x118
     0.34%   +0.49%  main+0x11c
     2.15%   -0.22%  main+0x120
     0.41%   +0.22%  main+0x123
     1.86%   +0.04%  main+0x12f
     3.86%   -0.69%  main+0x133
     0.02%           main+0x137
     3.80%   -1.19%  main+0x13d
     0.26%   +0.45%  main+0x141
     2.26%   +1.41%  main+0x145
     8.78%   -1.59%  main+0x148
     0.05%           main+0x14c
     1.40%   -0.26%  main+0x155
     0.07%           main+0x158
     0.74%   -0.03%  main+0x15b
     1.06%   -0.17%  main+0x160
     0.31%   +0.30%  main+0x1a8
     1.82%   -0.51%  main+0x1af
     0.09%   +0.07%  main+0x1b1
     0.05%           main+0x1b4
Quoted 4 lines

could we add something like '-s symstr' to do only symbol string comparison, so the previous output would gather in single line like:

     5.09%   +2.07%  main

Yes, we have a patch to do that.
https://lkml.org/lkml/2014/11/21/367
The default sort key "symbol" do symbols names comparison.

The symoff here is an extension in case anyone want to do deeper analysis.

Thanks,
Kan

Quoted 8 lines

otherwise the patchset looks ok to me:

thanks,
jirka

#5 lore raw

On Mon, Dec 01, 2014 at 08:05:48PM +0000, Liang, Kan wrote:

SNIP

Quoted 13 lines
Quoted 13 lines
     0.05%           main+0x14c
     1.40%   -0.26%  main+0x155
     0.07%           main+0x158
     0.74%   -0.03%  main+0x15b
     1.06%   -0.17%  main+0x160
     0.31%   +0.30%  main+0x1a8
     1.82%   -0.51%  main+0x1af
     0.09%   +0.07%  main+0x1b1
     0.05%           main+0x1b4

could we add something like '-s symstr' to do only symbol string comparison, so the previous output would gather in single line like:

Quoted 7 lines
Quoted 1 line
     5.09%   +2.07%  main

Yes, we have a patch to do that.
https://lkml.org/lkml/2014/11/21/367
The default sort key "symbol" do symbols names comparison.

The symoff here is an extension in case anyone want to do deeper analysis.

cool, I just acked it

thanks,
jirka