perf probe: Improve error message when function not found
PATCH 1/2 perf probe: Improve error message when function not found
When requesting a function from a userspace library the error message to the user is less than helpful. e.g.,
perf probe -x /lib64/libpthread-2.14.90.so -a 'lock_full=__pthread_mutex_lock_full' no symbols found in /lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to load map.
Error: Failed to add events. (-22)
In this cae the symbol really does exist but is a local symbol which is filtered:
nm /lib64/libpthread-2.14.90.so | grep __pthread_mutex_lock_full
0000000000005700 t __pthread_mutex_lock_fullWith this patch:
perf probe -x /lib64/libpthread-2.14.90.so -a 'lock_full=__pthread_mutex_lock_full'
no symbols found in /lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to find function in /lib64/libpthread-2.14.90.so. Perhaps it is a local variable?
Error: Failed to add events. (-22)
--- tools/perf/util/probe-event.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
tools/perf/util/probe-event.c +2 -1
@@ -2309,7 +2309,8 @@ static int convert_name_to_addr(struct perf_probe_event *pev, const char *exec) } available_func_filter =strfilter__new (function,NULL );if (map__load (map, filter_available_functions)) { -pr_err ("Failed to load map.\n" ); +pr_err ("Failed to find requested function in %s. Perhaps it is a local variable?\n" , + name); goto out; }
Signature · 1 line
1.8.3.4 (Apple Git-47)
PATCH 2/2 perf probe: Allow user to specify address within executable
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
--- tools/perf/Documentation/perf-probe.txt | 2 +- tools/perf/util/probe-event.c | 37 +++++++++++++++++++++++++-------- 2 files changed, 29 insertions(+), 10 deletions(-)
tools/perf/Documentation/perf-probe.txt +1 -1
@@ -123,7 +123,7 @@ Probe points are defined by following syntax. 'EVENT' specifies the name of new event, if omitted, it will be set the name of the probed function. Currently, event group name is set as 'probe'. -'FUNC' specifies a probed function name, and it may have one of the following options; '+OFFS' is the offset from function entry address in bytes, ':RLN' is the relative-line number from function entry line, and '%return' means that it probes function return. And ';PTN' means lazy matching pattern (see LAZY MATCHING). Note that ';PTN' must be the end of the probe point definition. In addition, '@SRC' specifies a source file which has that function. +'FUNC' specifies a probed function name, and it may have one of the following options; '+OFFS' is the offset from function entry address in bytes, ':RLN' is the relative-line number from function entry line, and '%return' means that it probes function return. And ';PTN' means lazy matching pattern (see LAZY MATCHING). Note that ';PTN' must be the end of the probe point definition. In addition, '@SRC' specifies a source file which has that function. If 'FUNC' starts with 0x and the entire string converts to an unsigned long it is treated as an address. It is also possible to specify a probe point by the source line number or lazy matching by using 'SRC:ALN' or 'SRC;PTN' syntax, where 'SRC' is the source file path, ':ALN' is the line number and ';PTN' is the lazy matching pattern. 'ARG' specifies the arguments of this probe point, (see PROBE ARGUMENT).
tools/perf/util/probe-event.c +28 -9
@@ -2208,6 +2208,9 @@ static struct strfilter *available_func_filter;static int filter_available_functions (struct map *map__maybe_unused ,struct symbol *sym) { +if (available_func_filter ==NULL ) +return 0 ; +if (sym->binding ==STB_GLOBAL &&strfilter__compare (available_func_filter, sym->name))return 0 ;
@@ -2281,9 +2284,10 @@ static int convert_name_to_addr(struct perf_probe_event *pev, const char *exec)struct perf_probe_point *pp = &pev->point;struct symbol *sym;struct map *map =NULL ; -char *function =NULL , *name =NULL ; +char *function =NULL , *name =NULL , *endptr;int ret = -EINVAL ;unsigned long long vaddr =0 ; +unsigned long fcn_addr =0 ;if (!pp->function) {pr_warning ("No function specified for uprobes" );
@@ -2297,6 +2301,13 @@ static int convert_name_to_addr(struct perf_probe_event *pev, const char *exec) goto out; } +/* perhaps an address was given instead of a function name */ +if (strncmp (function,"0x" ,2 ) ==0 ) { + fcn_addr =strtoul (function, &endptr,0 ); +if (endptr && *endptr !='\0' ) + fcn_addr =0 ; + } + name =realpath (exec,NULL );if (!name) {pr_warning ("Cannot find realpath for %s.\n" , exec);
@@ -2307,22 +2318,30 @@ static int convert_name_to_addr(struct perf_probe_event *pev, const char *exec)pr_warning ("Cannot find appropriate DSO for %s.\n" , exec); goto out; } - available_func_filter =strfilter__new (function,NULL ); + +if (fcn_addr ==0 ) + available_func_filter =strfilter__new (function,NULL ); +if (map__load (map, filter_available_functions)) {pr_err ("Failed to find requested function in %s. Perhaps it is a local variable?\n" , name); goto out; } - sym =map__find_symbol_by_name (map, function,NULL ); -if (!sym) { -pr_warning ("Cannot find %s in DSO %s\n" , function, exec); - goto out; +if (fcn_addr) { + vaddr = fcn_addr + pp->offset + map->pgoff; + }else { + sym =map__find_symbol_by_name (map, function,NULL ); +if (!sym) { +pr_warning ("Cannot find %s in DSO %s\n" , function, exec); + goto out; + } + +if (map->start > sym->start) + vaddr = map->start; + vaddr += sym->start + pp->offset + map->pgoff; } -if (map->start > sym->start) - vaddr = map->start; - vaddr += sym->start + pp->offset + map->pgoff; pp->offset =0 ;if (!pev->event) {
Signature · 1 line
1.8.3.4 (Apple Git-47)
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 4 lines · David Ahern
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
Hmm, IMHO, this kind of functionality is only good for debugging. In that case, you should use uprobe_events interface directly.
I recommend you to try enabing dwarf support in user space. Perf probe is a user-friendly interface of dynamic events.
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
In reply to David AhernQuoted 4 lines · David Ahern
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
Hmm, IMHO, this kind of functionality is only good for debugging. In that case, you should use uprobe_events interface directly.
I recommend you to try enabing dwarf support in user space. Perf probe is a user-friendly interface of dynamic events.
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · David Ahern
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
Hmm, IMHO, this kind of functionality is only good for debugging.
Why? perf-probe takes the function name, looks up the address in the executable, adds an offset and then pushes that address to the kernel. Why can't I have an option to jump straight to the desired address?
A few use cases that come to mind:
1. Allows manual creation of return probes in kernels which do not have that support.
2. probing static functions
3. probing versioned addresses in libc -- something the current syntax does not allow because '@' is used as a field separator:
$ nm /lib64/libc-2.14.90.so | grep pthread_cond_timedwait 00000000000fe6c0 t __pthread_cond_timedwait 000000000012d920 t __pthread_cond_timedwait_2_0 00000000000fe6c0 T pthread_cond_timedwait@@GLIBC_2.3.2 000000000012d920 T pthread_cond_timedwait@GLIBC_2.2.5
$ perf probe -x /lib64/libc-2.14.90.so -a 'cond_timewait=pthread_cond_timedwait' no symbols found in /lib64/libc-2.14.90.so, maybe install a debug package? Failed to load map. Error: Failed to add events. (-22)
$ perf probe -x /lib64/libc-2.14.90.so -a 'cond_timewait=pthread_cond_timedwait@@GLIBC_2.3.2' Semantic error :SRC@SRC is not allowed.
4. A workaround for any other shortcomings in the interface that require patches to fix and backports to be done. backports which take time if even possible.
Quoted 1 line
In that case, you should use
uprobe_eventsinterface directly.
How do I do that within the context of perf?
Quoted 3 lines
I recommend you to try enabing dwarf support in user space. Perf probe is a user-friendly interface of dynamic events.
I do have dwarf support enabled in perf. Are you referring to other components of userspace (like system libraries)?
David
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 13 lines · David Ahern
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · David Ahern
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
Hmm, IMHO, this kind of functionality is only good for debugging.
Why? perf-probe takes the function name, looks up the address in the executable, adds an offset and then pushes that address to the kernel. Why can't I have an option to jump straight to the desired address?
Quoted 6 lines
A few use cases that come to mind:
1. Allows manual creation of return probes in kernels which do not have that support.
2. probing static functions
D'oh. That is supposed to be local functions, not static functions. Right now perf-probe does not allow probes in local functions of an executable. Giving a specific address provides a mean to do that.
Quoted 38 lines
3. probing versioned addresses in libc -- something the current syntax does not allow because '@' is used as a field separator:
$ nm /lib64/libc-2.14.90.so | grep pthread_cond_timedwait 00000000000fe6c0 t __pthread_cond_timedwait 000000000012d920 t __pthread_cond_timedwait_2_0 00000000000fe6c0 T pthread_cond_timedwait@@GLIBC_2.3.2 000000000012d920 T pthread_cond_timedwait@GLIBC_2.2.5$ perf probe -x /lib64/libc-2.14.90.so -a 'cond_timewait=pthread_cond_timedwait' no symbols found in /lib64/libc-2.14.90.so, maybe install a debug package? Failed to load map. Error: Failed to add events. (-22)$ perf probe -x /lib64/libc-2.14.90.so -a 'cond_timewait=pthread_cond_timedwait@@GLIBC_2.3.2' Semantic error :SRC@SRC is not allowed.4. A workaround for any other shortcomings in the interface that require patches to fix and backports to be done. backports which take time if even possible.
Quoted 1 line
In that case, you should use
uprobe_eventsinterface directly.How do I do that within the context of perf?
Quoted 3 lines
I recommend you to try enabing dwarf support in user space. Perf probe is a user-friendly interface of dynamic events.
I do have dwarf support enabled in perf. Are you referring to other components of userspace (like system libraries)?
David
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 16 lines · David Ahern
Quoted 15 lines · David Ahern
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines · David Ahern
Allow user to specify an address within an executable. This is useful, for example, in probing local functions. If the function name begins with 0x then try to convert the supplied name to an address. If succuessful then treat the function name as the address within the executable to be probed.
Hmm, IMHO, this kind of functionality is only good for debugging.
Why? perf-probe takes the function name, looks up the address in the executable, adds an offset and then pushes that address to the kernel. Why can't I have an option to jump straight to the desired address?
A few use cases that come to mind:
Quoted 8 lines
Quoted 4 lines
1. Allows manual creation of return probes in kernels which do not have that support.
2. probing static functions
D'oh. That is supposed to be local functions, not static functions. Right now perf-probe does not allow probes in local functions of an executable. Giving a specific address provides a mean to do that.
No, perf probe for kprobe support to probing on local functions even if it is inlined. The problem currently you encountered is that perf probe uprobe support is incomplete. It should be enhanced to support dwarf(debuginfo) to find the functions(even lines) from it.
I don't want to make perf-probe just a wrapper of the ftrace dynamic event interface, because it doesn't add any "value" for users. Current uprobe support state is not sufficient for me.
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
In that case, you should use
uprobe_eventsinterface directly.How do I do that within the context of perf?
No way, but here, you can save this script as perf-uprobe and can use it for that purpose. :)
---
#!/bin/sh
BIN=$1
FUNC=$2
ADDR=nm $BIN | grep " $FUNC\$" | cut -f1 -d" " echo p $BIN:0x$ADDR > /sys/kernel/debug/tracing/uprobe_events
---
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
I recommend you to try enabing dwarf support in user space. Perf probe is a user-friendly interface of dynamic events.
I do have dwarf support enabled in perf. Are you referring to other components of userspace (like system libraries)?
Ah, right. Unfortunately, perf probe for uprobes does not support dwarf for user space yet. This should be enabled for completing the perf probe work...
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 2 lines · Masami Hiramatsu
I don't want to make perf-probe just a wrapper of the ftrace dynamic event interface, because it doesn't add any "value" for users.
Sure it does -- a consistent user experience in using a single command (perf probe ...) for setting up userspace probes.
Specifying the address directly is really on par with perf-record having the rNNN interface for directly passing the raw event id to the kernel.
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
In that case, you should use
uprobe_eventsinterface directly.How do I do that within the context of perf?
No way, but here, you can save this script as perf-uprobe and can use it for that purpose. :)
I figured out what you meant by uprobe_events interface yesterday. If I have to go to that interface for even 1 function I would do it for all -- from a user perspective it is just simpler to have 1 command to setup probes. I would prefer that 1 command be perf-probe.
David
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 9 lines · David Ahern
Quoted 2 lines · Masami Hiramatsu
I don't want to make perf-probe just a wrapper of the ftrace dynamic event interface, because it doesn't add any "value" for users.
Sure it does -- a consistent user experience in using a single command (perf probe ...) for setting up userspace probes.
Specifying the address directly is really on par with perf-record having the rNNN interface for directly passing the raw event id to the kernel.
No, since the perf interface is not directly exposed to users the perftools need to provide such raw interface. But the dynamic event has it on debugfs.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
In that case, you should use
uprobe_eventsinterface directly.How do I do that within the context of perf?
No way, but here, you can save this script as perf-uprobe and can use it for that purpose. :)
I figured out what you meant by
uprobe_eventsinterface yesterday. If I have to go to that interface for even 1 function I would do it for all -- from a user perspective it is just simpler to have 1 command to setup probes. I would prefer that 1 command be perf-probe.
Yeah, but in that case, why you don't ask us adding sym->binding == STB_LOCAL in filter_available_functions? :)
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 7 lines · Masami Hiramatsu
Quoted 4 lines
I figured out what you meant by
uprobe_eventsinterface yesterday. If I have to go to that interface for even 1 function I would do it for all -- from a user perspective it is just simpler to have 1 command to setup probes. I would prefer that 1 command be perf-probe.Yeah, but in that case, why you don't ask us adding
sym->binding==STB_LOCALinfilter_available_functions? :)
I did in a separate email -- you said because there can be multiple local functions with the same name. But local functions is not the only use case I need.
For now I will carry the patch locally. At this point I am 20 patches deep and have probably another 20 to go. What's one more. I'll come back to this when I have more time.
Thanks,
David
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 7 lines · David Ahern
Quoted 5 lines · Masami Hiramatsu
Quoted 4 lines
I figured out what you meant by
uprobe_eventsinterface yesterday. If I have to go to that interface for even 1 function I would do it for all -- from a user perspective it is just simpler to have 1 command to setup probes. I would prefer that 1 command be perf-probe.
Quoted 5 lines
Quoted 2 lines
Yeah, but in that case, why you don't ask us adding
sym->binding==STB_LOCALinfilter_available_functions? :)I did in a separate email -- you said because there can be multiple local functions with the same name.
Yeah, and this still seems to be a kind of workaround for me. The best way to make you requirement enable is to support dwarf for userspace tracing. OK I'll try it.
Quoted 2 lines
But local functions is not the only
use case I need.
What would you like to do with perf probe? Direct address accessing for userspace is not a good way to do, because userspace is relocatable...
Quoted 3 lines
For now I will carry the patch locally. At this point I am 20 patches deep and have probably another 20 to go. What's one more. I'll come back to this when I have more time.
Would you have any public git repository for that? And could you share us what would you like to do before sending patch? We can help you to tell the best way.
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Re: [PATCH 2/2] perf probe: Allow user to specify address within executable
Quoted 2 lines · Masami Hiramatsu
What would you like to do with perf probe? Direct address accessing for userspace is not a good way to do, because userspace is relocatable...
Masami, I am referring to the address within the executable which is what perf-probe pushes to the kernel.
Quoted 6 lines
Quoted 4 lines
For now I will carry the patch locally. At this point I am 20 patches deep and have probably another 20 to go. What's one more. I'll come back to this when I have more time.
Would you have any public git repository for that?
https://github.com/dsahern/linux/tree/perf-full-monty
And could you share
Quoted 2 lines
us what would you like to do before sending patch? We can help you to tell the best way.
I want to save myself some time and effort by giving the relative address with an executable. That allows me to probe any function, and I want to do it from a consistent user interface -- perf-probe. With the proposed patch I have that.
David
Quoted 11 lines · David Ahern
When requesting a function from a userspace library the error message to the user is less than helpful. e.g.,
perf probe -x
/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full' no symbols found in/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to load map.
Error: Failed to add events. (-22)In this cae the symbol really does exist but is a local symbol which is filtered:
Quoted 8 lines
nm /lib64/libpthread-2.14.90.so | grep __pthread_mutex_lock_full 0000000000005700 t __pthread_mutex_lock_fullWith this patch:
perf probe -x/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full'no symbols found in
/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to find function in/lib64/libpthread-2.14.90.so. Perhaps it is a local variable?
Hmm, indeed current message is too less information, but your message looks wired. I think just showing "Failed to find requested symbol in %s" is enough. :)
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Quoted 15 lines · Masami Hiramatsu
Quoted 14 lines · David Ahern
When requesting a function from a userspace library the error message to the user is less than helpful. e.g.,
perf probe -x
/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full' no symbols found in/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to load map.
Error: Failed to add events. (-22)In this cae the symbol really does exist but is a local symbol which is filtered:
nm /lib64/libpthread-2.14.90.so | grep __pthread_mutex_lock_full 0000000000005700 t __pthread_mutex_lock_full
Quoted 8 lines
Quoted 5 lines
With this patch:
perf probe -x/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full'no symbols found in
/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to find function in/lib64/libpthread-2.14.90.so. Perhaps it is a local variable?Hmm, indeed current message is too less information, but your message looks wired. I think just showing "Failed to find requested symbol in %s" is enough. :)
I disagree. perf-probe filters non-global variables:
if (sym->binding == STB_GLOBAL &&
strfilter__compare(available_func_filter, sym->name))
return 0;That needs to be explicitly stated in the error message -- only global symbols may be given.
Between the 2 original error messages I wasted 2 hours strace'ing perf looking at files -- including all the debuginfo variants -- that perf opens. nm/objdump and every other tool sees the symbols yet perf throws out bizarre error messages -- "no symbols found in ..." and "Failed to load map". Let's fix those messages with something meaningful.
David
Quoted 25 lines · David Ahern
Quoted 15 lines · Masami Hiramatsu
Quoted 14 lines · David Ahern
When requesting a function from a userspace library the error message to the user is less than helpful. e.g.,
perf probe -x
/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full' no symbols found in/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to load map.
Error: Failed to add events. (-22)In this cae the symbol really does exist but is a local symbol which is filtered:
nm /lib64/libpthread-2.14.90.so | grep __pthread_mutex_lock_full 0000000000005700 t __pthread_mutex_lock_fullQuoted 8 lines
Quoted 5 lines
With this patch:
perf probe -x/lib64/libpthread-2.14.90.so-a 'lock_full=__pthread_mutex_lock_full'no symbols found in
/lib64/libpthread-2.14.90.so, maybe install a debug package? Failed to find function in/lib64/libpthread-2.14.90.so. Perhaps it is a local variable?Hmm, indeed current message is too less information, but your message looks wired. I think just showing "Failed to find requested symbol in %s" is enough. :)
Quoted 8 lines
I disagree. perf-probe filters non-global variables:
if (sym->binding == STB_GLOBAL && strfilter__compare(available_func_filter, sym->name)) return 0;That needs to be explicitly stated in the error message -- only global symbols may be given.
Ah, I see. In that case, I think the "variable" is not a correct word, the "symbol" is better, because perf probe can take tracing variable arguments after the trace point definition. :)
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]
Quoted 6 lines · Masami Hiramatsu
Quoted 2 lines
That needs to be explicitly stated in the error message -- only global symbols may be given.
Ah, I see. In that case, I think the "variable" is not a correct word, the "symbol" is better, because perf probe can take tracing variable arguments after the trace point definition. :)
Right, variable is not the correct word. So how about this:
"Failed to find requested symbol in %s. Is it a global symbol?"
David
Quoted 11 lines · David Ahern
Quoted 6 lines · Masami Hiramatsu
Quoted 2 lines
That needs to be explicitly stated in the error message -- only global symbols may be given.
Ah, I see. In that case, I think the "variable" is not a correct word, the "symbol" is better, because perf probe can take tracing variable arguments after the trace point definition. :)
Right, variable is not the correct word. So how about this:
"Failed to find requested symbol in %s. Is it a global symbol?"
Yeah, that's good for me now :)
Thank you,
Signature · 4 lines
Masami HIRAMATSU IT Management Research Dept. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory E-mail: [email protected]