Skip to contentSkip to conversation

problems with perf probe and system libraries

PATCH In review David Ahern 4 participants 6 messages Dec 30, 2014 – Jan 8, 2015 perf events

lore
Share & tools
lore
#0 lore raw

Hi Masami:

I have been looking at perf-probe again and having a number of problems with top of tree.

Here's the first one I have isolated:

$ perf probe -x /lib64/libc-2.18.so -a 'malloc  size=%di'
Probe point 'malloc' not found.
  Error: Failed to add events.
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc
malloc
malloc@plt
malloc_atfork
malloc_check
malloc_consolidate
malloc_hook_ini
malloc_info
malloc_printerr
mallochook
ptmalloc_init
ptmalloc_init.part.8
ptmalloc_lock_all
ptmalloc_unlock_all
ptmalloc_unlock_all2
tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
Probe point 'malloc' not found.
  Error: Failed to add events.

A year ago (v3.12) this worked fine so I did a git bisect which points to:

commit fb7345bbf7fad9bf72ef63a19c707970b9685812

Author: Masami Hiramatsu <[email protected]>
Date:   Thu Dec 26 05:41:53 2013 +0000

perf probe: Support basic dwarf-based operations on uprobe events

I have tried top of tree on Fedora 16, 18 and 20 with a variety of kernels - and a variety of results. Reverting to 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this one and it works again.

David

#1 lore raw

Hi David,

Quoted 10 lines · David Ahern

Hi Masami:

I have been looking at perf-probe again and having a number of problems with top of tree.

Here's the first one I have isolated:

$ perf probe -x /lib64/libc-2.18.so -a 'malloc  size=%di'
Probe point 'malloc' not found.
   Error: Failed to add events.

It seems to fail when failed to open debuginfo.
I confirmed it on ubuntu 14.04 too.
I'll try to fix that.

Thank you for reporting!

Quoted 42 lines
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc
malloc
malloc@plt
malloc_atfork
malloc_check
malloc_consolidate
malloc_hook_ini
malloc_info
malloc_printerr
mallochook
ptmalloc_init
ptmalloc_init.part.8
ptmalloc_lock_all
ptmalloc_unlock_all
ptmalloc_unlock_all2
tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di'
Probe point 'malloc' not found.
   Error: Failed to add events.

A year ago (v3.12) this worked fine so I did a git bisect which points to:

commit fb7345bbf7fad9bf72ef63a19c707970b9685812

Author: Masami Hiramatsu <[email protected]>
Date:   Thu Dec 26 05:41:53 2013 +0000

perf probe: Support basic dwarf-based operations on uprobe events

I have tried top of tree on Fedora 16, 18 and 20 with a variety of kernels - and a variety of results. Reverting to 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this one and it works again.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#2 lore raw

PATCH 1/1 perf-probe: Fix to fall back to find probe point in symbols

In reply to David Ahern

Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.

This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.

David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355

I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6

             name                 (strp) "malloc"
             decl_file            (data1) 25
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  81b5]
             declaration          (flag_present)
 [  8f58]      formal_parameter
 --
             name                 (strp) "malloc"
             decl_file            (data1) 23
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  9f4a]
             declaration          (flag_present)
             sibling              (ref4) [  bb29]

...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.

However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$

  1181: 0000000000080700   5332 FUNC    LOCAL  DEFAULT       12 _int_malloc
  4537: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __GI___libc_malloc
  5545: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __malloc
  6063: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 malloc
  7302: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 __libc_malloc

-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.

To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.

---
 tools/perf/util/probe-event.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index 28eb141..7f9b863 100644
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found.\n",
+		pr_warning("Probe point '%s' not found in debuginfo.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		return -ENOENT;
+		if (need_dwarf)
+			return -ENOENT;
+		return 0;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
maintainer Acked-by on v1 1/1
#3 lore raw

Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols

Hi Masami,

Quoted 40 lines · Masami Hiramatsu

Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.

This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.

David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355

I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6

             name                 (strp) "malloc"
             decl_file            (data1) 25
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  81b5]
             declaration          (flag_present)
 [  8f58]      formal_parameter
 --
             name                 (strp) "malloc"
             decl_file            (data1) 23
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  9f4a]
             declaration          (flag_present)
             sibling              (ref4) [  bb29]

...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.

However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$

  1181: 0000000000080700   5332 FUNC    LOCAL  DEFAULT       12 _int_malloc
  4537: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __GI___libc_malloc
  5545: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __malloc
  6063: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 malloc
  7302: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 __libc_malloc
Quoted 7 lines

-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.

To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.

Idealy, it'd be great if it can go to the definition. I think perf only keeps a single symbol at a same address so if there're multiple aliases at a specific address it might be missed even with this patch.

But that will be a rare case, so

Thanks,
Namhyung

Quoted 31 lines
---
 tools/perf/util/probe-event.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -3
index 28eb141..7f9b863 100644
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found.\n",
+		pr_warning("Probe point '%s' not found in debuginfo.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		return -ENOENT;
+		if (need_dwarf)
+			return -ENOENT;
+		return 0;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw

Re: Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols

Quoted 48 lines · Namhyung Kim

Hi Masami,

Quoted 45 lines · Masami Hiramatsu

Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.

This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.

David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355

I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6

             name                 (strp) "malloc"
             decl_file            (data1) 25
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  81b5]
             declaration          (flag_present)
 [  8f58]      formal_parameter
 --
             name                 (strp) "malloc"
             decl_file            (data1) 23
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  9f4a]
             declaration          (flag_present)
             sibling              (ref4) [  bb29]

...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.

However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$

  1181: 0000000000080700   5332 FUNC    LOCAL  DEFAULT       12 _int_malloc
  4537: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __GI___libc_malloc
  5545: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __malloc
  6063: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 malloc
  7302: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 __libc_malloc

-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.

Quoted 6 lines
Quoted 2 lines

To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.

Idealy, it'd be great if it can go to the definition. I think perf only keeps a single symbol at a same address so if there're multiple aliases at a specific address it might be missed even with this patch.

Hmm, anyway, that should be fixed by improving perf's map, instead of introducing new symbol finder.

Quoted 3 lines

But that will be a rare case, so

Thank you!

Quoted 42 lines

Thanks,
Namhyung

Quoted 27 lines
---
 tools/perf/util/probe-event.c |    6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -3
index 28eb141..7f9b863 100644
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found.\n",
+		pr_warning("Probe point '%s' not found in debuginfo.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		return -ENOENT;
+		if (need_dwarf)
+			return -ENOENT;
+		return 0;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);
 
--
Quoted 4 lines

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 4 lines
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]
#5 lore raw

[tip:perf/urgent] perf probe: Fix to fall back to find probe point in symbols

In reply to Masami Hiramatsu
perf probe: Fix to fall back to find probe point in symbols
Commit-ID:  906451b98b6774042b707a5dcebb6a93dbd14c85
Gitweb:     http://git.kernel.org/tip/906451b98b6774042b707a5dcebb6a93dbd14c85
Author:     Masami Hiramatsu <[email protected]>
AuthorDate: Wed, 31 Dec 2014 15:27:47 +0900
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Fri, 2 Jan 2015 12:42:17 -0300

perf probe: Fix to fall back to find probe point in symbols

Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.

This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.

David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355

I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6

             name                 (strp) "malloc"
             decl_file            (data1) 25
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  81b5]
             declaration          (flag_present)
 [  8f58]      formal_parameter
 --
             name                 (strp) "malloc"
             decl_file            (data1) 23
             decl_line            (data2) 466
             prototyped           (flag_present)
             type                 (ref4) [  9f4a]
             declaration          (flag_present)
             sibling              (ref4) [  bb29]

...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.

However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$

  1181: 0000000000080700   5332 FUNC    LOCAL  DEFAULT       12 _int_malloc
  4537: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __GI___libc_malloc
  5545: 00000000000831d0    339 FUNC    LOCAL  DEFAULT       12 __malloc
  6063: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 malloc
  7302: 00000000000831d0    339 FUNC    GLOBAL DEFAULT       12 __libc_malloc

-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.

To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.

---
 tools/perf/util/probe-event.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
index 28eb141..7f9b863 100644
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
 	}
 
 	if (ntevs == 0)	{	/* No error but failed to find probe point. */
-		pr_warning("Probe point '%s' not found.\n",
+		pr_warning("Probe point '%s' not found in debuginfo.\n",
 			   synthesize_perf_probe_point(&pev->point));
-		return -ENOENT;
+		if (need_dwarf)
+			return -ENOENT;
+		return 0;
 	}
 	/* Error path : ntevs < 0 */
 	pr_debug("An error occurred in debuginfo analysis (%d).\n", ntevs);