perf-probe: improve glibc support

PATCH Superseded Masami Hiramatsu 4 participants 14 messages Mar 2, 2015 – Mar 4, 2015 perf events

lore

Share & tools

lore

Masami Hiramatsu

author

2015-03-02 12:50 UTC #0 lore raw

cover perf-probe: improve glibc support

Hi,

Here is a series of patches which improves perf-probe to handle glibc's aliased symbols more correctly. Longly, a major known issue of probing on glibc is that the some aliased symbols(e.g. malloc) can not find by perf-probe.

Actually, glibc's malloc symbol is just an alias of __libc_malloc. Its debuginfo knows only __libc_malloc, and perf's symbol map knows only malloc. This difference always confuses users that they can see malloc by perf report or annotate, but they can not probe on it, nor find definitions by --line option.

Previously, I've made a commit 906451b98b67 which solved this problem partly, but not completely fixed. So I decided to solve this issue completely by finding the symbols like malloc from perf's symbol map, and converting the symbol's address into debuginfo's location infomation.

With this series, you can use --vars, --line and --add with the aliased symbols on glibc.

Note that the 1/4 patch is just a resend patch which I sent last weekend (since this series depends on that)
http://www.gossamer-threads.com/lists/linux/kernel/2115623

2/4 and 3/4 introduces above logic to solve this issue. And 4/4 reverts previous incomplete fix.

Thank you,

---

Masami Hiramatsu (4):
[RESEND][BUGFIX] perf-probe: Remove bias offset to find probe point by address
perf-probe: Fix to handle aliased symbols in glibc
perf-probe: Fix --line to handle aliased symbols in glibc
Revert "perf probe: Fix to fall back to find probe point in symbols"

 tools/perf/util/probe-event.c  |  181 +++++++++++++++++++++++++++++++++++-----
 tools/perf/util/probe-finder.c |    5 -
 2 files changed, 160 insertions(+), 26 deletions(-)

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Masami Hiramatsu

author

2015-03-02 12:51 UTC #1 lore raw

PATCH RESEND 1/4 perf-probe: Remove bias offset to find probe point by address

Remove bias offset to find probe point by address.

Without this patch, probe points on kernel and executables are shown correctly, but not work with libraries.
-----

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
  -----

Removing bias allows to show it as real place.
-----

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)
  -----

Signed-off-by Masami Hiramatsu <[email protected]>

---
 tools/perf/util/probe-finder.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

tools/perf/util/probe-finder.c +1 -4

index d141935..46f009a 100644
@@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr,
 	const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp;
 	int baseline = 0, lineno = 0, ret = 0;
 
-	/* Adjust address with bias */
-	addr += dbg->bias;
-
 	/* Find cu die */
-	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) {
+	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) {
 		pr_warning("Failed to find debug information for address %lx\n",
 			   addr);
 		ret = -EINVAL;

tip-bot for Masami Hiramatsu

merge bot

2015-03-03 06:27 UTC #2 lore raw

[tip:perf/core] perf probe: Remove bias offset to find probe point by address

perf probe: Remove bias offset to find probe point by address

Commit-ID:  0104fe69e0287cf3635657b4c6b26a18e0091697
Gitweb:     http://git.kernel.org/tip/0104fe69e0287cf3635657b4c6b26a18e0091697
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Mon, 2 Mar 2015 21:49:46 +0900
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Mon, 2 Mar 2015 12:34:38 -0300

perf probe: Remove bias offset to find probe point by address

Remove bias offset to find probe point by address.

Without this patch, probe points on kernel and executables are shown correctly, but do not work with libraries:

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)

Removing bias allows it to show it as real place:

  # ./perf probe -l
    probe:do_fork        (on do_fork@kernel/fork.c)
    probe_libc:malloc    (on __libc_malloc@malloc/malloc.c in /usr/lib64/libc-2.17.so)
    probe_perf:strlist__new (on strlist__new@util/strlist.c in /home/mhiramat/ksrc/linux-3/tools/perf/perf)

Signed-off-by Masami Hiramatsu <[email protected]>

Cc David Ahern <[email protected]>

Cc Jiri Olsa <[email protected]>

Cc Namhyung Kim <[email protected]>

Cc Naohiro Aota <[email protected]>

Cc Peter Zijlstra <[email protected]>

Link http://lkml.kernel.org/r/[email protected] open here

Signed-off-by Arnaldo Carvalho de Melo <[email protected]>

---
 tools/perf/util/probe-finder.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

tools/perf/util/probe-finder.c +1 -4

index d141935..46f009a 100644
@@ -1345,11 +1345,8 @@ int debuginfo__find_probe_point(struct debuginfo *dbg, unsigned long addr,
 	const char *fname = NULL, *func = NULL, *basefunc = NULL, *tmp;
 	int baseline = 0, lineno = 0, ret = 0;
 
-	/* Adjust address with bias */
-	addr += dbg->bias;
-
 	/* Find cu die */
-	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr - dbg->bias, &cudie)) {
+	if (!dwarf_addrdie(dbg->dbg, (Dwarf_Addr)addr, &cudie)) {
 		pr_warning("Failed to find debug information for address %lx\n",
 			   addr);
 		ret = -EINVAL;

Masami Hiramatsu

author

2015-03-02 12:51 UTC #3 lore raw

PATCH 2/4 perf-probe: Fix to handle aliased symbols in glibc

In reply to Masami Hiramatsu

Fix perf probe to handle aliased symbols correctly in glibc. In the glibc, several symbols are defined as an alias of __libc_XXX, e.g. malloc is an alias of __libc_malloc. In such cases, dwarf has no subroutine instances of the alias functions (e.g. no "malloc" instance), but the map has that symbol and its address.
Thus, if we search the alieased symbol in debuginfo, we always fail to find it, but it is in the map.

To solve this problem, this fails back to address-based alternative search, which searches the symbol in the map, translates its address to alternative (correct) function name by using debuginfo, and retry to find the alternative function point from debuginfo.

This adds fail-back process to --vars, --lines and --add options. So, now you can use those on malloc@libc :)

Without this patch;
-----

  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Failed to find the address of malloc
    Error: Failed to show vars.
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Probe point 'malloc' not found in debuginfo.
    Error: Failed to add events.
  -----

With this patch;
-----

  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)

You can now use it in all perf tools, such as:

perf record -e probe_libc:malloc -aR sleep 1
-----

Reported-by Arnaldo Carvalho de Melo <[email protected]>

Signed-off-by Masami Hiramatsu <[email protected]>

---
 tools/perf/util/probe-event.c |  140 ++++++++++++++++++++++++++++++++++++-----
 1 file changed, 124 insertions(+), 16 deletions(-)

tools/perf/util/probe-event.c +124 ~~-16~~

index 1c570c2fa7..b8f4578 100644
@@ -178,6 +178,25 @@ static struct map *kernel_get_module_map(const char *module)
 	return NULL;
 }
 
+static struct map *get_target_map(const char *target, bool user)
+{
+	/* Init maps of given executable or kernel */
+	if (user)
+		return dso__new_map(target);
+	else
+		return kernel_get_module_map(target);
+}
+
+static void put_target_map(struct map *map, bool user)
+{
+	if (map && user) {
+		/* Only the user map needs to be released */
+		dso__delete(map->dso);
+		map__delete(map);
+	}
+}
+
+
 static struct dso *kernel_get_module_dso(const char *module)
 {
 	struct dso *dso;

@@ -249,6 +268,13 @@ out:
 	return ret;
 }
 
+static void clear_perf_probe_point(struct perf_probe_point *pp)
+{
+	free(pp->file);
+	free(pp->function);
+	free(pp->lazy_line);
+}
+
 static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
 {
 	int i;

@@ -258,6 +284,74 @@ static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
 }
 
 #ifdef HAVE_DWARF_SUPPORT
+/*
+ * Some binaries like glibc have special symbols which are on the symbol
+ * table, but not in the debuginfo. If we can find the address of the
+ * symbol from map, we can translate the address back to the probe point.
+ */
+static int find_alternative_probe_point(struct debuginfo *dinfo,
+					struct perf_probe_point *pp,
+					struct perf_probe_point *result,
+					const char *target, bool uprobes)
+{
+	struct map *map = NULL;
+	struct symbol *sym;
+	u64 address = 0;
+	int ret = -ENOENT;
+
+	/* This can work only for function-name based one */
+	if (!pp->function || pp->file)
+		return -ENOTSUP;
+
+	map = get_target_map(target, uprobes);
+	if (!map)
+		return -EINVAL;
+
+	/* Find the address of given function */
+	map__for_each_symbol_by_name(map, pp->function, sym) {
+		if (sym->binding == STB_GLOBAL || sym->binding == STB_LOCAL) {
+			address = sym->start;
+			break;
+		}
+	}
+	if (!address) {
+		ret = -ENOENT;
+		goto out;
+	}
+	pr_debug("Symbol %s address found : %lx\n", pp->function, address);
+
+	ret = debuginfo__find_probe_point(dinfo, (unsigned long)address,
+					  result);
+	if (ret <= 0)
+		ret = (!ret) ? -ENOENT : ret;
+	else {
+		result->offset += pp->offset;
+		result->line += pp->line;
+		ret = 0;
+	}
+
+out:
+	put_target_map(map, uprobes);
+	return ret;
+
+}
+
+static int get_alternative_probe_event(struct debuginfo *dinfo,
+				       struct perf_probe_event *pev,
+				       struct perf_probe_point *tmp,
+				       const char *target)
+{
+	int ret;
+
+	memcpy(tmp, &pev->point, sizeof(*tmp));
+	memset(&pev->point, 0, sizeof(pev->point));
+	ret = find_alternative_probe_point(dinfo, tmp, &pev->point,
+					   target, pev->uprobes);
+	if (ret < 0)
+		memcpy(&pev->point, tmp, sizeof(*tmp));
+
+	return ret;
+}
 
 /* Open new debuginfo of given module */
 static struct debuginfo *open_debuginfo(const char *module, bool silent)

@@ -466,6 +560,7 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 					  int max_tevs, const char *target)
 {
 	bool need_dwarf = perf_probe_event_need_dwarf(pev);
+	struct perf_probe_point tmp;
 	struct debuginfo *dinfo;
 	int ntevs, ret = 0;

@@ -482,6 +577,20 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	/* Searching trace events corresponding to a probe event */
 	ntevs = debuginfo__find_trace_events(dinfo, pev, tevs, max_tevs);
 
+	if (ntevs == 0)	{  /* Not found, retry with an alternative */
+		ret = get_alternative_probe_event(dinfo, pev, &tmp, target);
+		if (!ret) {
+			ntevs = debuginfo__find_trace_events(dinfo, pev,
+							     tevs, max_tevs);
+			/*
+			 * Write back to the original probe_event for
+			 * setting appropriate (user given) event name
+			 */
+			clear_perf_probe_point(&pev->point);
+			memcpy(&pev->point, &tmp, sizeof(tmp));
+		}
+	}
+
 	debuginfo__delete(dinfo);
 
 	if (ntevs > 0) {	/* Succeeded to find trace events */

@@ -719,12 +828,13 @@ int show_line_range(struct line_range *lr, const char *module, bool user)
 static int show_available_vars_at(struct debuginfo *dinfo,
 				  struct perf_probe_event *pev,
 				  int max_vls, struct strfilter *_filter,
-				  bool externs)
+				  bool externs, const char *target)
 {
 	char *buf;
 	int ret, i, nvars;
 	struct str_node *node;
 	struct variable_list *vls = NULL, *vl;
+	struct perf_probe_point tmp;
 	const char *var;
 
 	buf = synthesize_perf_probe_point(&pev->point);

@@ -734,6 +844,15 @@ static int show_available_vars_at(struct debuginfo *dinfo,
 
 	ret = debuginfo__find_available_vars_at(dinfo, pev, &vls,
 						max_vls, externs);
+	if (!ret) {  /* Not found, retry with an alternative */
+		ret = get_alternative_probe_event(dinfo, pev, &tmp, target);
+		if (!ret) {
+			ret = debuginfo__find_available_vars_at(dinfo, pev,
+						&vls, max_vls, externs);
+			/* Release the old probe_point */
+			clear_perf_probe_point(&tmp);
+		}
+	}
 	if (ret <= 0) {
 		if (ret == 0 || ret == -ENOENT) {
 			pr_err("Failed to find the address of %s\n", buf);

@@ -796,7 +915,7 @@ int show_available_vars(struct perf_probe_event *pevs, int npevs,
 
 	for (i = 0; i < npevs && ret >= 0; i++)
 		ret = show_available_vars_at(dinfo, &pevs[i], max_vls, _filter,
-					     externs);
+					     externs, module);
 
 	debuginfo__delete(dinfo);
 out:

@@ -1742,15 +1861,12 @@ static int convert_to_perf_probe_event(struct probe_trace_event *tev,
 
 void clear_perf_probe_event(struct perf_probe_event *pev)
 {
-	struct perf_probe_point *pp = &pev->point;
 	struct perf_probe_arg_field *field, *next;
 	int i;
 
 	free(pev->event);
 	free(pev->group);
-	free(pp->file);
-	free(pp->function);
-	free(pp->lazy_line);
+	clear_perf_probe_point(&pev->point);
 
 	for (i = 0; i < pev->nargs; i++) {
 		free(pev->args[i].name);

@@ -2367,11 +2483,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
 	int num_matched_functions;
 	int ret, i;
 
-	/* Init maps of given executable or kernel */
-	if (pev->uprobes)
-		map = dso__new_map(target);
-	else
-		map = kernel_get_module_map(target);
+	map = get_target_map(target, pev->uprobes);
 	if (!map) {
 		ret = -EINVAL;
 		goto out;

@@ -2464,11 +2576,7 @@ static int find_probe_trace_events_from_map(struct perf_probe_event *pev,
 	}
 
 out:
-	if (map && pev->uprobes) {
-		/* Only when using uprobe(exec) map needs to be released */
-		dso__delete(map->dso);
-		map__delete(map);
-	}
+	put_target_map(map, pev->uprobes);
 	return ret;
 
 nomem_out:

Arnaldo Carvalho de Melo

maintainer

2015-03-02 15:46 UTC #4 lore raw

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:

Quoted 10 lines

With this patch;
-----

  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)

Quoted 5 lines

You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1
Reported-by Arnaldo Carvalho de Melo <[email protected]>

Humm, not working for me, after the patch:

[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
[root@ssdandy ~]#

And then the one asking for 'bytes' to be collectd fails.

After processing the other patches I'll try to debug this...

[root@ssdandy ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc

   438: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 __libc_malloc@@GLIBC_2.2.5
   545: 0000000000082320   239 FUNC    GLOBAL DEFAULT   12 malloc_info@@GLIBC_2.10
   810: 00000000000820c0   490 FUNC    WEAK   DEFAULT   12 malloc_stats@@GLIBC_2.2.5
   981: 00000000000802e0   507 FUNC    WEAK   DEFAULT   12 malloc_get_state@@GLIBC_2.2.5
  1077: 00000000003ba740     8 OBJECT  WEAK   DEFAULT   32 __malloc_hook@@GLIBC_2.2.5
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
  1204: 0000000000080d30   222 FUNC    WEAK   DEFAULT   12 malloc_usable_size@@GLIBC_2.2.5
  1450: 0000000000081d50   604 FUNC    WEAK   DEFAULT   12 malloc_trim@@GLIBC_2.2.5
  1767: 00000000003bca60     8 OBJECT  WEAK   DEFAULT   33 __malloc_initialize_hook@@GLIBC_2.2.5
  2061: 00000000000814f0  1286 FUNC    WEAK   DEFAULT   12 malloc_set_state@@GLIBC_2.2.5
    95: 00000000003bbaa0     4 OBJECT  LOCAL  DEFAULT   33 cache_malloced
  1004: 0000000000000000     0 FILE    LOCAL  DEFAULT  ABS malloc.c
  1005: 000000000007b060   275 FUNC    LOCAL  DEFAULT   12 ptmalloc_lock_all
  1011: 00000000003bcb30     8 OBJECT  LOCAL  DEFAULT   33 save_malloc_hook
  1013: 00000000000801c0   285 FUNC    LOCAL  DEFAULT   12 malloc_atfork
  1017: 000000000007b180   158 FUNC    LOCAL  DEFAULT   12 ptmalloc_unlock_all2
  1030: 000000000007ba90   174 FUNC    LOCAL  DEFAULT   12 ptmalloc_unlock_all
  1033: 000000000007bb40   232 FUNC    LOCAL  DEFAULT   12 malloc_printerr
  1057: 000000000007c3d0  1518 FUNC    LOCAL  DEFAULT   12 malloc_consolidate
  1089: 000000000007dee0  5195 FUNC    LOCAL  DEFAULT   12 _int_malloc
  1100: 000000000007f330   282 FUNC    LOCAL  DEFAULT   12 malloc_check
  1117: 00000000003bca90     4 OBJECT  LOCAL  DEFAULT   33 disallow_malloc_check
  1118: 00000000003bcaa0     4 OBJECT  LOCAL  DEFAULT   33 using_malloc_checking
  1136: 0000000000080fe0  1015 FUNC    LOCAL  DEFAULT   12 ptmalloc_init.part.8
  1138: 00000000000813e0    21 FUNC    LOCAL  DEFAULT   12 ptmalloc_init
  1139: 0000000000081400    60 FUNC    LOCAL  DEFAULT   12 malloc_hook_ini
  1160: 0000000000082fc0   495 FUNC    LOCAL  DEFAULT   12 mallochook
  1162: 00000000003bcbf0     8 OBJECT  LOCAL  DEFAULT   33 old_malloc_hook
  1181: 00000000003bcc40     8 OBJECT  LOCAL  DEFAULT   33 tr_old_malloc_hook
  1182: 0000000000083fc0   189 FUNC    LOCAL  DEFAULT   12 tr_mallochook
  1194: 00000000003bcc60     8 OBJECT  LOCAL  DEFAULT   33 malloc_trace_buffer
  3673: 00000000003ba170     4 OBJECT  LOCAL  DEFAULT   32 __libc_malloc_initialized
  3734: 00000000000814f0  1286 FUNC    LOCAL  DEFAULT   12 __malloc_set_state
  4047: 0000000000080d30   222 FUNC    LOCAL  DEFAULT   12 __malloc_usable_size
  4101: 0000000000081d50   604 FUNC    LOCAL  DEFAULT   12 __malloc_trim
  4338: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __GI___libc_malloc
  4531: 00000000000802e0   507 FUNC    LOCAL  DEFAULT   12 __malloc_get_state
  4569: 00000000000820c0   490 FUNC    LOCAL  DEFAULT   12 __malloc_stats
  4849: 0000000000080050   107 FUNC    LOCAL  DEFAULT   12 __malloc_check_init
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  5490: 00000000003bca60     8 OBJECT  WEAK   DEFAULT   33 __malloc_initialize_hook
  5571: 00000000000814f0  1286 FUNC    WEAK   DEFAULT   12 malloc_set_state
  5868: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc
  5878: 0000000000082320   239 FUNC    GLOBAL DEFAULT   12 malloc_info
  5988: 0000000000081d50   604 FUNC    WEAK   DEFAULT   12 malloc_trim
  6526: 00000000003ba740     8 OBJECT  WEAK   DEFAULT   32 __malloc_hook
  6615: 0000000000080d30   222 FUNC    WEAK   DEFAULT   12 malloc_usable_size
  7087: 00000000000802e0   507 FUNC    WEAK   DEFAULT   12 malloc_get_state
  7104: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 __libc_malloc
  7271: 00000000000820c0   490 FUNC    WEAK   DEFAULT   12 malloc_stats

[acme@ssdandy linux]$

- Arnaldo

Masami Hiramatsu

author

2015-03-03 02:39 UTC #5 lore raw

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Quoted 15 lines · Arnaldo Carvalho de Melo

Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
Quoted 13 lines
With this patch;
-----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1

Quoted 8 lines

Quoted 1 line
Reported-by Arnaldo Carvalho de Melo <[email protected]>
Humm, not working for me, after the patch:
[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)

Could you run it with -v (verbose) option?

Quoted 11 lines

[root@ssdandy ~]#
And then the one asking for 'bytes' to be collectd fails.
After processing the other patches I'll try to debug this...
[root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64

^^^ why is this different from the glibc version??

Quoted 2 lines

[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc

[...]

Quoted 1 line

4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init

__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0

Quoted 2 lines

  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5

that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it.

Thank you,

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Arnaldo Carvalho de Melo

maintainer

2015-03-03 02:45 UTC #6 lore raw

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:

Quoted 16 lines

Quoted 15 lines · Arnaldo Carvalho de Melo
Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
Quoted 13 lines
With this patch;
-----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1

Quoted 9 lines

Quoted 8 lines
Quoted 1 line
Reported-by Arnaldo Carvalho de Melo <[email protected]>
Humm, not working for me, after the patch:
[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)

Will try after a 'make build-test' finishes for the current batch

Quoted 20 lines

Quoted 11 lines
[root@ssdandy ~]#
And then the one asking for 'bytes' to be collectd fails.
After processing the other patches I'll try to debug this...
[root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64
^^^ why is this different from the glibc version??
Quoted 2 lines
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
[...]
Quoted 1 line
4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init
__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0

Quoted 7 lines

Quoted 2 lines
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it.

This one?

commit 0104fe69e0287cf3635657b4c6b26a18e0091697

Author: Masami Hiramatsu <[email protected]>
Date:   Mon Mar 2 21:49:46 2015 +0900

perf probe: Remove bias offset to find probe point by address

Remove bias offset to find probe point by address.

----------------

Yes, it is applied.

- Arnaldo

Arnaldo Carvalho de Melo

maintainer

2015-03-03 03:05 UTC #7 lore raw

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:

Quoted 21 lines

Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
Quoted 20 lines
Quoted 19 lines · Arnaldo Carvalho de Melo
Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
Quoted 10 lines
With this patch;
-----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
Quoted 5 lines
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1
Reported-by Arnaldo Carvalho de Melo <[email protected]>
Humm, not working for me, after the patch:

Quoted 7 lines

Quoted 5 lines
Quoted 4 lines
[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
Will try after a 'make build-test' finishes for the current batch

[root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug Searching variables at malloc
Symbol malloc address found : 800c0
Get 2611 lines from this CU
Probe point found: __malloc_check_init+96
Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
[root@ssdandy ~]#

If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is what is there for the malloc routines:

[root@ssdandy ~]# grep malloc /tmp/4 probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) Searching variables at malloc
symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
symbol__new: malloc_atfork 0x801c0-0x802dd
symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
symbol__new: malloc_printerr 0x7bb40-0x7bc28
symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
symbol__new: _int_malloc 0x7dee0-0x7f32b
symbol__new: malloc_check 0x7f330-0x7f44a
symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
symbol__new: ptmalloc_init 0x813e0-0x813f5
symbol__new: malloc_hook_ini 0x81400-0x8143c
symbol__new: mallochook 0x82fc0-0x831af
symbol__new: tr_mallochook 0x83fc0-0x8407d
symbol__new: __malloc_set_state 0x814f0-0x819f6
symbol__new: __malloc_usable_size 0x80d30-0x80e0e
symbol__new: __malloc_trim 0x81d50-0x81fac
symbol__new: __GI___libc_malloc 0x800c0-0x801b5
symbol__new: __malloc_get_state 0x802e0-0x804db
symbol__new: __malloc_stats 0x820c0-0x822aa
symbol__new: __malloc_check_init 0x80050-0x800bb
symbol__new: __malloc 0x800c0-0x801b5
symbol__new: malloc_set_state 0x814f0-0x819f6
symbol__new: malloc 0x800c0-0x801b5
symbol__new: malloc_info 0x82320-0x8240f
symbol__new: malloc_trim 0x81d50-0x81fac
symbol__new: malloc_usable_size 0x80d30-0x80e0e
symbol__new: malloc_get_state 0x802e0-0x804db
symbol__new: __libc_malloc 0x800c0-0x801b5
symbol__new: malloc_stats 0x820c0-0x822aa
symbol__new: malloc@plt 0x1f300-0x1f310
Symbol malloc address found : 800c0
Probe point found: __malloc_check_init+96 [root@ssdandy ~]#

Quoted 43 lines

Quoted 20 lines
Quoted 11 lines
[root@ssdandy ~]#
And then the one asking for 'bytes' to be collectd fails.
After processing the other patches I'll try to debug this...
[root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64
^^^ why is this different from the glibc version??
Quoted 2 lines
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
[...]
Quoted 1 line
4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init
__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
Quoted 7 lines
Quoted 2 lines
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it.
This one?
commit 0104fe69e0287cf3635657b4c6b26a18e0091697
Author: Masami Hiramatsu <[email protected]>
Date:   Mon Mar 2 21:49:46 2015 +0900
perf probe: Remove bias offset to find probe point by address
Remove bias offset to find probe point by address.
----------------
Yes, it is applied.
- Arnaldo

Masami Hiramatsu

author

2015-03-03 04:11 UTC #8 lore raw

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Quoted 30 lines · Arnaldo Carvalho de Melo

Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 28 lines
Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
Quoted 16 lines
Quoted 15 lines · Arnaldo Carvalho de Melo
Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
Quoted 13 lines
With this patch;
-----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
Reported-by Arnaldo Carvalho de Melo <[email protected]>
Humm, not working for me, after the patch:
[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
Will try after a 'make build-test' finishes for the current batch

Thank you for checking this,

Quoted 8 lines

[root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug Searching variables at malloc
Symbol malloc address found : 800c0
Get 2611 lines from this CU

Hmm, something wrong at here.

Quoted 1 line

Probe point found: __malloc_check_init+96

This seems that the debuginfo is a bit odd. Could you also run eu-addr2line for 0x800c0 and 0x800b0?

I'm using CentOS7 with a bit newer glibc rpms.

# rpm -q glibc glibc-debuginfo
glibc-2.17-55.el7_0.5.x86_64
glibc-debuginfo-2.17-55.el7_0.5.x86_64

And I got following results.

[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 __libc_malloc /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855 [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 __malloc_check_init /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75

So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

Thank you,

Quoted 7 lines

Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
[root@ssdandy ~]#
If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is what is there for the malloc routines:

Yeah, it seems symbol maps works fine. Debuginfo analysis failed.

Quoted 88 lines

[root@ssdandy ~]# grep malloc /tmp/4 probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) Searching variables at malloc
symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
symbol__new: malloc_atfork 0x801c0-0x802dd
symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
symbol__new: malloc_printerr 0x7bb40-0x7bc28
symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
symbol__new: _int_malloc 0x7dee0-0x7f32b
symbol__new: malloc_check 0x7f330-0x7f44a
symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
symbol__new: ptmalloc_init 0x813e0-0x813f5
symbol__new: malloc_hook_ini 0x81400-0x8143c
symbol__new: mallochook 0x82fc0-0x831af
symbol__new: tr_mallochook 0x83fc0-0x8407d
symbol__new: __malloc_set_state 0x814f0-0x819f6
symbol__new: __malloc_usable_size 0x80d30-0x80e0e
symbol__new: __malloc_trim 0x81d50-0x81fac
symbol__new: __GI___libc_malloc 0x800c0-0x801b5
symbol__new: __malloc_get_state 0x802e0-0x804db
symbol__new: __malloc_stats 0x820c0-0x822aa
symbol__new: __malloc_check_init 0x80050-0x800bb
symbol__new: __malloc 0x800c0-0x801b5
symbol__new: malloc_set_state 0x814f0-0x819f6
symbol__new: malloc 0x800c0-0x801b5
symbol__new: malloc_info 0x82320-0x8240f
symbol__new: malloc_trim 0x81d50-0x81fac
symbol__new: malloc_usable_size 0x80d30-0x80e0e
symbol__new: malloc_get_state 0x802e0-0x804db
symbol__new: __libc_malloc 0x800c0-0x801b5
symbol__new: malloc_stats 0x820c0-0x822aa
symbol__new: malloc@plt 0x1f300-0x1f310
Symbol malloc address found : 800c0
Probe point found: __malloc_check_init+96 [root@ssdandy ~]#
Quoted 36 lines
Quoted 20 lines
Quoted 11 lines
[root@ssdandy ~]#
And then the one asking for 'bytes' to be collectd fails.
After processing the other patches I'll try to debug this...
[root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64
^^^ why is this different from the glibc version??
Quoted 2 lines
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
[...]
Quoted 1 line
4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init
__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
Quoted 7 lines
Quoted 2 lines
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it.
This one?
commit 0104fe69e0287cf3635657b4c6b26a18e0091697
Author: Masami Hiramatsu <[email protected]>
Date:   Mon Mar 2 21:49:46 2015 +0900
perf probe: Remove bias offset to find probe point by address
Quoted 7 lines
Remove bias offset to find probe point by address.
----------------
Yes, it is applied.
- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Arnaldo Carvalho de Melo

maintainer

2015-03-03 04:24 UTC #9 lore raw

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Em Tue, Mar 03, 2015 at 01:11:17PM +0900, Masami Hiramatsu escreveu:

Quoted 58 lines

Quoted 30 lines · Arnaldo Carvalho de Melo
Em Mon, Mar 02, 2015 at 11:45:12PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 28 lines
Em Tue, Mar 03, 2015 at 11:39:02AM +0900, Masami Hiramatsu escreveu:
Quoted 16 lines
Quoted 15 lines · Arnaldo Carvalho de Melo
Em Mon, Mar 02, 2015 at 09:49:53PM +0900, Masami Hiramatsu escreveu:
Quoted 13 lines
With this patch;
-----
  # ./perf probe -x /usr/lib64/libc-2.17.so -V malloc
  Available variables at malloc
          @<__libc_malloc+0>
                  size_t  bytes
  # ./perf probe -x /usr/lib64/libc-2.17.so -a "malloc bytes"
  Added new event:
    probe_libc:malloc    (on malloc in /usr/lib64/libc-2.17.so with bytes)
You can now use it in all perf tools, such as:
perf record -e probe_libc:malloc -aR sleep 1
Quoted 9 lines
Quoted 8 lines
Quoted 1 line
Reported-by Arnaldo Carvalho de Melo <[email protected]>
Humm, not working for me, after the patch:
[root@ssdandy ~]# perf probe -x /usr/lib64/libc-2.17.so -V malloc Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
Will try after a 'make build-test' finishes for the current batch
Thank you for checking this,
Quoted 8 lines
[root@ssdandy ~]# perf probe -vvv -x /usr/lib64/libc-2.17.so -V malloc probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Open Debuginfo file: /usr/lib/debug/usr/lib64/libc-2.17.so.debug Searching variables at malloc
Symbol malloc address found : 800c0
Get 2611 lines from this CU
Hmm, something wrong at here.
Quoted 1 line
Probe point found: __malloc_check_init+96
This seems that the debuginfo is a bit odd. Could you also run eu-addr2line for 0x800c0 and 0x800b0?
I'm using CentOS7 with a bit newer glibc rpms.
# rpm -q glibc glibc-debuginfo
glibc-2.17-55.el7_0.5.x86_64
glibc-debuginfo-2.17-55.el7_0.5.x86_64
And I got following results.

Quoted 8 lines

[mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 __libc_malloc /usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:2855 [mhiramat@localhost perf]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 __malloc_check_init /usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
So, 0x800b0 correctly points __malloc_check_init+96 but its address is not 0x800c0.

[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800c0 malloc
??:0
[acme@ssdandy linux]$ eu-addr2line -fi -e /usr/lib64/libc-2.17.so 0x800b0 __malloc_check_init
??:0
[acme@ssdandy linux]$

If I do it over the debuginfo files:

[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug 0x800c0 __malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:75
[acme@ssdandy linux]$ addr2line -fi -e /usr/lib/debug/usr/lib64/libc-2.17.so.debug 0x800b0 __malloc_check_init
/usr/src/debug/glibc-2.17-c758a686/malloc/hooks.c:82
[acme@ssdandy linux]$

Tomorrow I'll check for newer packages or any reports about problems with debuginfo files, please let me know if you need any further info,

Off to bed, zzZzz

- Arnaldo

Quoted 109 lines

Thank you,
Quoted 7 lines
Available variables at malloc
@<__malloc_check_init+96>
(No matched variables)
[root@ssdandy ~]#
If I add one more 'v' I get the symtabs as read by symbol-elf.c and this is what is there for the malloc routines:
Yeah, it seems symbol maps works fine. Debuginfo analysis failed.
Quoted 88 lines
[root@ssdandy ~]# grep malloc /tmp/4 probe-definition(0): malloc
symbol:malloc file:(null) line:0 offset:0 return:0 lazy:(null) Searching variables at malloc
symbol__new: ptmalloc_lock_all 0x7b060-0x7b173
symbol__new: malloc_atfork 0x801c0-0x802dd
symbol__new: ptmalloc_unlock_all2 0x7b180-0x7b21e
symbol__new: ptmalloc_unlock_all 0x7ba90-0x7bb3e
symbol__new: malloc_printerr 0x7bb40-0x7bc28
symbol__new: malloc_consolidate 0x7c3d0-0x7c9be
symbol__new: _int_malloc 0x7dee0-0x7f32b
symbol__new: malloc_check 0x7f330-0x7f44a
symbol__new: ptmalloc_init.part.8 0x80fe0-0x813d7
symbol__new: ptmalloc_init 0x813e0-0x813f5
symbol__new: malloc_hook_ini 0x81400-0x8143c
symbol__new: mallochook 0x82fc0-0x831af
symbol__new: tr_mallochook 0x83fc0-0x8407d
symbol__new: __malloc_set_state 0x814f0-0x819f6
symbol__new: __malloc_usable_size 0x80d30-0x80e0e
symbol__new: __malloc_trim 0x81d50-0x81fac
symbol__new: __GI___libc_malloc 0x800c0-0x801b5
symbol__new: __malloc_get_state 0x802e0-0x804db
symbol__new: __malloc_stats 0x820c0-0x822aa
symbol__new: __malloc_check_init 0x80050-0x800bb
symbol__new: __malloc 0x800c0-0x801b5
symbol__new: malloc_set_state 0x814f0-0x819f6
symbol__new: malloc 0x800c0-0x801b5
symbol__new: malloc_info 0x82320-0x8240f
symbol__new: malloc_trim 0x81d50-0x81fac
symbol__new: malloc_usable_size 0x80d30-0x80e0e
symbol__new: malloc_get_state 0x802e0-0x804db
symbol__new: __libc_malloc 0x800c0-0x801b5
symbol__new: malloc_stats 0x820c0-0x822aa
symbol__new: malloc@plt 0x1f300-0x1f310
Symbol malloc address found : 800c0
Probe point found: __malloc_check_init+96 [root@ssdandy ~]#
Quoted 36 lines
Quoted 20 lines
Quoted 11 lines
[root@ssdandy ~]#
And then the one asking for 'bytes' to be collectd fails.
After processing the other patches I'll try to debug this...
[root@ssdandy ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.0 (Maipo) [root@ssdandy ~]# rpm -q glibc glibc-debuginfo glibc-2.17-55.el7_0.3.x86_64 glibc-debuginfo-2.17-55.el7_0.1.x86_64
^^^ why is this different from the glibc version??
Quoted 2 lines
[root@ssdandy ~]#
[acme@ssdandy linux]$ readelf -Ws /usr/lib64/libc-2.17.so| grep malloc
[...]
Quoted 1 line
4849: 0000000000080050 107 FUNC LOCAL DEFAULT 12 __malloc_check_init
__malloc_check_init+96(0x60) becomes 0x80050 + 0x60 = 0x800b0
Quoted 7 lines
Quoted 2 lines
  5351: 00000000000800c0   245 FUNC    LOCAL  DEFAULT   12 __malloc
  1170: 00000000000800c0   245 FUNC    GLOBAL DEFAULT   12 malloc@@GLIBC_2.2.5
that is not 0x800c0, so something goes wrong when translating the address to probe point. Could you check the patch 1/4 was applied? Actually there is a bug in the routine which gets the probe point from address. 1/4 fixes it.
This one?
commit 0104fe69e0287cf3635657b4c6b26a18e0091697
Author: Masami Hiramatsu <[email protected]>
Date:   Mon Mar 2 21:49:46 2015 +0900
perf probe: Remove bias offset to find probe point by address
Quoted 7 lines
Remove bias offset to find probe point by address.
----------------
Yes, it is applied.
- Arnaldo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Namhyung Kim

maintainer

2015-03-03 13:32 UTC #10 lore raw

Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

In reply to Masami Hiramatsu

Hi Masami,

Quoted 9 lines · Masami Hiramatsu

Fix perf probe to handle aliased symbols correctly in glibc. In the glibc, several symbols are defined as an alias of __libc_XXX, e.g. malloc is an alias of __libc_malloc. In such cases, dwarf has no subroutine instances of the alias functions (e.g. no "malloc" instance), but the map has that symbol and its address.
Thus, if we search the alieased symbol in debuginfo, we always fail to find it, but it is in the map.

Quoted 8 lines

To solve this problem, this fails back to address-based alternative search, which searches the symbol in the map, translates its address to alternative (correct) function name by using debuginfo, and retry to find the alternative function point from debuginfo.
This adds fail-back process to --vars, --lines and --add options. So, now you can use those on malloc@libc :)

So this is only for binaries that have debuginfo, right?

I have a similar issue with no debuginfo.

  $ perf probe -x /usr/lib/libc.so.6 -V calloc
  The /usr/lib/libc-2.21.so file has no debug information.
  Rebuild with -g, or install an appropriate debuginfo package.
    Error: Failed to show vars.

But it also failed to add a probe to calloc:

  $ perf probe -x /usr/lib/libc.so.6 -a calloc
  Failed to find symbol calloc in /usr/lib/libc-2.21.so
    Error: Failed to add events.

Of course there's calloc in the libc binary.

  $ nm /usr/lib/libc.so.6 | grep calloc
  000000000007b1f0 t __calloc
  000000000007b1f0 T __libc_calloc
  000000000007b1f0 W calloc

I think the problem is that calloc is a weak symbol so it'll be discarded during the symbol loading. It's because to avoid multiple symbols (or aliases) at a same address so we choose a better symbol using heuristics. But for this case I think we can allow aliases since it's used only for finding probe points.

Thanks,
Namhyung

Masami Hiramatsu

author

2015-03-04 05:48 UTC #11 lore raw

Re: Re: [PATCH perf/core 2/4] perf-probe: Fix to handle aliased symbols in glibc

Quoted 40 lines · Namhyung Kim

Hi Masami,
Quoted 9 lines · Masami Hiramatsu
Fix perf probe to handle aliased symbols correctly in glibc. In the glibc, several symbols are defined as an alias of __libc_XXX, e.g. malloc is an alias of __libc_malloc. In such cases, dwarf has no subroutine instances of the alias functions (e.g. no "malloc" instance), but the map has that symbol and its address.
Thus, if we search the alieased symbol in debuginfo, we always fail to find it, but it is in the map.
Quoted 8 lines
To solve this problem, this fails back to address-based alternative search, which searches the symbol in the map, translates its address to alternative (correct) function name by using debuginfo, and retry to find the alternative function point from debuginfo.
This adds fail-back process to --vars, --lines and --add options. So, now you can use those on malloc@libc :)
So this is only for binaries that have debuginfo, right?
I have a similar issue with no debuginfo.
  $ perf probe -x /usr/lib/libc.so.6 -V calloc
  The /usr/lib/libc-2.21.so file has no debug information.
  Rebuild with -g, or install an appropriate debuginfo package.
    Error: Failed to show vars.
But it also failed to add a probe to calloc:
  $ perf probe -x /usr/lib/libc.so.6 -a calloc
  Failed to find symbol calloc in /usr/lib/libc-2.21.so
    Error: Failed to add events.
Of course there's calloc in the libc binary.

Quoted 8 lines

  $ nm /usr/lib/libc.so.6 | grep calloc
  000000000007b1f0 t __calloc
  000000000007b1f0 T __libc_calloc
  000000000007b1f0 W calloc
I think the problem is that calloc is a weak symbol so it'll be discarded during the symbol loading.

Right, I also hit same problem on waitpid, which is also an weak symbol.

Quoted 4 lines

It's because to avoid multiple
symbols (or aliases) at a same address so we choose a better symbol using heuristics. But for this case I think we can allow aliases since it's used only for finding probe points.

I'm just using struct map and dso, so those should be improved.

Thank you,

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Masami Hiramatsu

author

2015-03-02 12:51 UTC #12 lore raw

PATCH 3/4 perf-probe: Fix --line to handle aliased symbols in glibc

In reply to Masami Hiramatsu

Fix perf probe --line to handle aliased symbols correctly in glibc.

This makes line_range search failing back to address-based alternative search as same as --add and --vars.

Without this patch;
-----

  # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
  Specified source line is not found.
    Error: Failed to show lines.
  -----

With this patch;
-----

  # ./perf probe -x /usr/lib64/libc-2.17.so -L malloc
  <__libc_malloc@/usr/src/debug/glibc-2.17-c758a686/malloc/malloc.c:0>
        0  __libc_malloc(size_t bytes)
        1  {
             mstate ar_ptr;
             void *victim;

             __malloc_ptr_t (*hook) (size_t, const __malloc_ptr_t)
        6      = force_reg (__malloc_hook);
        7    if (__builtin_expect (hook != NULL, 0))
        8      return (*hook)(bytes, RETURN_ADDRESS (0));

       10    arena_lookup(ar_ptr);

       12    arena_lock(ar_ptr, bytes);

-----

Note that this actually shows __libc_malloc, since it is the real instance of malloc. User can use both __libc_malloc and malloc for --line.

Signed-off-by Masami Hiramatsu <[email protected]>

---
 tools/perf/util/probe-event.c |   35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

tools/perf/util/probe-event.c +33 -2

index b8f4578..4cfd121 100644
@@ -353,6 +353,31 @@ static int get_alternative_probe_event(struct debuginfo *dinfo,
 	return ret;
 }
 
+static int get_alternative_line_range(struct debuginfo *dinfo,
+				      struct line_range *lr,
+				      const char *target, bool user)
+{
+	struct perf_probe_point pp = { 0 }, result = { 0 };
+	int ret, len = 0;
+
+	pp.function = lr->function;
+	pp.file = lr->file;
+	pp.line = lr->start;
+	if (lr->end != INT_MAX)
+		len = lr->end - lr->start;
+	ret = find_alternative_probe_point(dinfo, &pp, &result,
+					   target, user);
+	if (!ret) {
+		lr->function = result.function;
+		lr->file = result.file;
+		lr->start = result.line;
+		if (lr->end != INT_MAX)
+			lr->end = lr->start + len;
+		clear_perf_probe_point(&pp);
+	}
+	return ret;
+}
+
 /* Open new debuginfo of given module */
 static struct debuginfo *open_debuginfo(const char *module, bool silent)
 {

@@ -734,7 +759,8 @@ static int _show_one_line(FILE *fp, int l, bool skip, bool show_num)
  * Show line-range always requires debuginfo to find source file and
  * line number.
  */
-static int __show_line_range(struct line_range *lr, const char *module)
+static int __show_line_range(struct line_range *lr, const char *module,
+			     bool user)
 {
 	int l = 1;
 	struct int_node *ln;

@@ -750,6 +776,11 @@ static int __show_line_range(struct line_range *lr, const char *module)
 		return -ENOENT;
 
 	ret = debuginfo__find_line_range(dinfo, lr);
+	if (!ret) {	/* Not found, retry with an alternative */
+		ret = get_alternative_line_range(dinfo, lr, module, user);
+		if (!ret)
+			ret = debuginfo__find_line_range(dinfo, lr);
+	}
 	debuginfo__delete(dinfo);
 	if (ret == 0 || ret == -ENOENT) {
 		pr_warning("Specified source line is not found.\n");

@@ -819,7 +850,7 @@ int show_line_range(struct line_range *lr, const char *module, bool user)
 	ret = init_symbol_maps(user);
 	if (ret < 0)
 		return ret;
-	ret = __show_line_range(lr, module);
+	ret = __show_line_range(lr, module, user);
 	exit_symbol_maps();
 
 	return ret;

Masami Hiramatsu

author

2015-03-02 12:51 UTC #13 lore raw

PATCH 4/4 Revert "perf probe: Fix to fall back to find probe point in symbols"

In reply to Masami Hiramatsu

This reverts commit 906451b98b67 ("perf probe: Fix to fall back to find probe point in symbols").

Since perf-probe retries with the address of given symbol searched from map before this path, this fall back routine doesn't need anymore.

Signed-off-by Masami Hiramatsu <[email protected]>

---
 tools/perf/util/probe-event.c |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

tools/perf/util/probe-event.c +2 -4

index 4cfd121..c379ea0 100644
@@ -630,11 +630,9 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found in debuginfo.\n",
+		pr_warning("Probe point '%s' not found.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		if (need_dwarf)
-			return -ENOENT;
-		return 0;
+		return -ENOENT;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);