problems with perf probe and system libraries
Hi Masami:
I have been looking at perf-probe again and having a number of problems with top of tree.
Here's the first one I have isolated:
$ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di' Probe point 'malloc' not found. Error: Failed to add events.
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc malloc malloc@plt malloc_atfork malloc_check malloc_consolidate malloc_hook_ini malloc_info malloc_printerr mallochook ptmalloc_init ptmalloc_init.part.8 ptmalloc_lock_all ptmalloc_unlock_all ptmalloc_unlock_all2 tr_mallochook
$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di' Probe point 'malloc' not found. Error: Failed to add events.
A year ago (v3.12) this worked fine so I did a git bisect which points to:
commit fb7345bbf7fad9bf72ef63a19c707970b9685812
Author: Masami Hiramatsu <[email protected]> Date: Thu Dec 26 05:41:53 2013 +0000
perf probe: Support basic dwarf-based operations on uprobe events
I have tried top of tree on Fedora 16, 18 and 20 with a variety of kernels - and a variety of results. Reverting to 8a613d40e389b723fd5889ac8d4033ed4030be31 which is the commit before this one and it works again.
David
Hi David,
Quoted 10 lines · David Ahern
Hi Masami:
I have been looking at perf-probe again and having a number of problems with top of tree.
Here's the first one I have isolated:
$ perf probe -x /lib64/libc-2.18.so -a 'malloc size=%di' Probe point 'malloc' not found. Error: Failed to add events.
It seems to fail when failed to open debuginfo.
I confirmed it on ubuntu 14.04 too.
I'll try to fix that.
Thank you for reporting!
Quoted 42 lines
$ perf probe -x /lib64/libc-2.18.so -F | grep malloc malloc malloc@plt malloc_atfork malloc_check malloc_consolidate malloc_hook_ini malloc_info malloc_printerr mallochook ptmalloc_init ptmalloc_init.part.8 ptmalloc_lock_all ptmalloc_unlock_all ptmalloc_unlock_all2 tr_mallochook$ perf probe -x /lib64/libc-2.18.so -a 'malloc=malloc size=%di' Probe point 'malloc' not found. Error: Failed to add events.A year ago (v3.12) this worked fine so I did a git bisect which points to:
commit
fb7345bbf7fad9bf72ef63a19c707970b9685812Author: Masami Hiramatsu <[email protected]> Date: Thu Dec 26 05:41:53 2013 +0000perf probe: Support basic dwarf-based operations on uprobe events
I have tried top of tree on Fedora 16, 18 and 20 with a variety of kernels - and a variety of results. Reverting to
8a613d40e389b723fd5889ac8d4033ed4030be31which is the commit before this one and it works again.David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
PATCH 1/1 perf-probe: Fix to fall back to find probe point in symbols
In reply to David AhernFix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.
This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
name (strp) "malloc"
decl_file (data1) 25
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 81b5]
declaration (flag_present)
[ 8f58] formal_parameter
--
name (strp) "malloc"
decl_file (data1) 23
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 9f4a]
declaration (flag_present)
sibling (ref4) [ bb29] ...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.
However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.
To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.
--- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, }if (ntevs ==0 ) {/* No error but failed to find probe point. */ -pr_warning ("Probe point '%s' not found.\n" , +pr_warning ("Probe point '%s' not found in debuginfo.\n" ,synthesize_perf_probe_point (&pev->point)); -return -ENOENT ; +if (need_dwarf) +return -ENOENT ; +return 0 ; }/* Error path : ntevs < 0 */ pr_debug ("An error occurred in debuginfo analysis (%d).\n" , ntevs);
Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols
Hi Masami,
Quoted 40 lines · Masami Hiramatsu
Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.
This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so| grep \"malloc\" -A6name (strp) "malloc" decl_file (data1) 25 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 81b5] declaration (flag_present) [ 8f58] formal_parameter -- name (strp) "malloc" decl_file (data1) 23 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 9f4a] declaration (flag_present) sibling (ref4) [ bb29]...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.However, there are some malloc instances in symbols.
-----
eu-readelf --symbols/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so| grep malloc$1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
Quoted 7 lines
-----
As you an see, malloc and__libc_mallochave same address, and actually__libc_mallochas an entry in debuginfo. So you can set up a probe on__libc_malloc.To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.
Idealy, it'd be great if it can go to the definition. I think perf only keeps a single symbol at a same address so if there're multiple aliases at a specific address it might be missed even with this patch.
But that will be a rare case, so
Thanks,
Namhyung
Quoted 31 lines
Signed-off-by Masami Hiramatsu <[email protected]>Reported-by David Ahern <[email protected]>--- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)tools/perf/util/probe-event.c +4
-3@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, }if (ntevs ==0 ) {/* No error but failed to find probe point. */ -pr_warning ("Probe point '%s' not found.\n" , +pr_warning ("Probe point '%s' not found in debuginfo.\n" ,synthesize_perf_probe_point (&pev->point)); -return -ENOENT ; +if (need_dwarf) +return -ENOENT ; +return 0 ; }/* Error path : ntevs < 0 */ pr_debug ("An error occurred in debuginfo analysis (%d).\n" , ntevs); --To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: Re: [PATCH] [BUGFIX] perf-probe: Fix to fall back to find probe point in symbols
Quoted 48 lines · Namhyung Kim
Hi Masami,
Quoted 45 lines · Masami Hiramatsu
Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.
This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so| grep \"malloc\" -A6name (strp) "malloc" decl_file (data1) 25 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 81b5] declaration (flag_present) [ 8f58] formal_parameter -- name (strp) "malloc" decl_file (data1) 23 decl_line (data2) 466 prototyped (flag_present) type (ref4) [ 9f4a] declaration (flag_present) sibling (ref4) [ bb29]...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.However, there are some malloc instances in symbols.
-----
eu-readelf --symbols/usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so| grep malloc$1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc-----
As you an see, malloc and__libc_mallochave same address, and actually__libc_mallochas an entry in debuginfo. So you can set up a probe on__libc_malloc.
Quoted 6 lines
Quoted 2 lines
To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.
Idealy, it'd be great if it can go to the definition. I think perf only keeps a single symbol at a same address so if there're multiple aliases at a specific address it might be missed even with this patch.
Hmm, anyway, that should be fixed by improving perf's map, instead of introducing new symbol finder.
Quoted 3 lines
But that will be a rare case, so
Acked-by Namhyung Kim <[email protected]>
Thank you!
Quoted 42 lines
Thanks,
NamhyungQuoted 27 lines
Signed-off-by Masami Hiramatsu <[email protected]>Reported-by David Ahern <[email protected]>--- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)tools/perf/util/probe-event.c +4
-3@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, }if (ntevs ==0 ) {/* No error but failed to find probe point. */ -pr_warning ("Probe point '%s' not found.\n" , +pr_warning ("Probe point '%s' not found in debuginfo.\n" ,synthesize_perf_probe_point (&pev->point)); -return -ENOENT ; +if (need_dwarf) +return -ENOENT ; +return 0 ; }/* Error path : ntevs < 0 */ pr_debug ("An error occurred in debuginfo analysis (%d).\n" , ntevs); --Quoted 4 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 4 lines
Masami HIRAMATSU Software Platform Research Dept. Linux Technology Research Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
tip-bot for Masami Hiramatsu
merge bot[tip:perf/urgent] perf probe: Fix to fall back to find probe point in symbols
In reply to Masami Hiramatsuperf probe: Fix to fall back to find probe point in symbols
Commit-ID: 906451b98b6774042b707a5dcebb6a93dbd14c85 Gitweb: http://git.kernel.org/tip/906451b98b6774042b707a5dcebb6a93dbd14c85 Author: Masami Hiramatsu <[email protected]> AuthorDate: Wed, 31 Dec 2014 15:27:47 +0900 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Fri, 2 Jan 2015 12:42:17 -0300
perf probe: Fix to fall back to find probe point in symbols
Fix to fall back to find a probe point in symbols if perf fails to find it in debuginfo.
This can happen when the target function is an alias of another function. Such alias doesn't have an entry in debuginfo but in symbols.
David Ahern reported this problem in https://lkml.org/lkml/2014/12/29/355
I ensured the problem and deeper investigation discovers it.
-----
eu-readelf --debug-dump=info /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep \"malloc\" -A6
name (strp) "malloc"
decl_file (data1) 25
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 81b5]
declaration (flag_present)
[ 8f58] formal_parameter
--
name (strp) "malloc"
decl_file (data1) 23
decl_line (data2) 466
prototyped (flag_present)
type (ref4) [ 9f4a]
declaration (flag_present)
sibling (ref4) [ bb29] ...
-----
All these entires have no instances (all of them are declarations) This is why the perf probe failed to find it in debuginfo.
However, there are some malloc instances in symbols.
-----
eu-readelf --symbols /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.19.so | grep malloc$
1181: 0000000000080700 5332 FUNC LOCAL DEFAULT 12 _int_malloc 4537: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __GI___libc_malloc 5545: 00000000000831d0 339 FUNC LOCAL DEFAULT 12 __malloc 6063: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 malloc 7302: 00000000000831d0 339 FUNC GLOBAL DEFAULT 12 __libc_malloc
-----
As you an see, malloc and __libc_malloc have same address, and actually __libc_malloc has an entry in debuginfo. So you can set up a probe on __libc_malloc.
To fix this problem shortly, perf probe simply falls back to find probe point(malloc) in symbols if it is not found in debuginfo.
--- tools/perf/util/probe-event.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
tools/perf/util/probe-event.c +4 -2
@@ -495,9 +495,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev, }if (ntevs ==0 ) {/* No error but failed to find probe point. */ -pr_warning ("Probe point '%s' not found.\n" , +pr_warning ("Probe point '%s' not found in debuginfo.\n" ,synthesize_perf_probe_point (&pev->point)); -return -ENOENT ; +if (need_dwarf) +return -ENOENT ; +return 0 ; }/* Error path : ntevs < 0 */ pr_debug ("An error occurred in debuginfo analysis (%d).\n" , ntevs);