Skip to contentSkip to conversation

failed to add c++ probe

DISCUSSION Jiri Olsa 4 participants 7 messages Sep 19, 2016 – Sep 23, 2016

lore
Share & tools
lore

Jiri Olsa

author
#0 lore raw

hi,
Thomas reported failure to add an uprobe on libstdc++.so.6 like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 'std::ostream::flush' Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

it does not work also if I use the mangled name like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' Probe point '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' not found.
Error: Failed to add events.

thanks,
jirka

#1 lore raw
Quoted 6 lines · On Mon, 19 Sep 2016 11:54:02 +0200 Jiri Olsa <[email protected]> wrote:

hi,
Thomas reported failure to add an uprobe on libstdc++.so.6 like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 'std::ostream::flush' Semantic error :There is non-digit char in line number.
Error: Command Parse Error.

Ah, I should start adding mangling support on perf probe...

Quoted 5 lines

it does not work also if I use the mangled name like:

[jolsa@krava perf]$ sudo ./perf probe -x /usr/lib64/libstdc++.so.6 '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' Probe point '_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_' not found.
Error: Failed to add events.

Hmm, could you try to find mangled symbol from the library as below?

./perf probe -x /usr/lib64/libstdc++.so.6 -F \*

And also try to do adding probe with -v option again?

In my case, I could find the mangled symbol, but failed to setup the probe...

$ sudo ./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_
probe-definition(0): _ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ 
symbol:_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Could not open debuginfo. Try to use symbols.
Opening /sys/kernel/debug/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_ZSt5flushIwSt11char_traitsIwEERSt13basic_ostreamIT_T0_ES6_ /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21:0x1114c0
Failed to write event: Invalid argument
  Error: Failed to add events. Reason: Invalid argument (Code: -22)

dmsg -k said;

[143725.339609] Failed to allocate trace_uprobe.(-22)
[143725.339612] Failed to parse address or file.

It seems we need to fix trace_uprobe.c at least.

Thank you!

Signature · 1 line
Masami Hiramatsu <[email protected]>
#2 lore raw

Masami,

Quoted 7 lines · Masami Hiramatsu

Hmm, could you try to find mangled symbol from the library as below?

./perf probe -x /usr/lib64/libstdc++.so.6 -F \*

And also try to do adding probe with -v option again?

In my case, I could find the mangled symbol, but failed to setup the probe...

./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
Try to find probe point from debuginfo.
Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
Matched function: is_open
found inline addr: 0x8cad0
Probe point found: is_open+0
found inline addr: 0x8cac0
Probe point found: is_open+0
found inline addr: 0x8cab0
Probe point found: is_open+0
Matched function: is_open
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

[16568.176464] Failed to allocate trace_uprobe.(-22)
[16568.176468] Failed to parse address or file.

Aside of that mangled symbols with a @plt suffix do not work at all.

Thanks,

tglx

#3 lore raw
Quoted 36 lines · On Tue, 20 Sep 2016 01:32:04 +0200 (CEST) Thomas Gleixner <[email protected]>…

Masami,

Quoted 7 lines · Masami Hiramatsu

Hmm, could you try to find mangled symbol from the library as below?

./perf probe -x /usr/lib64/libstdc++.so.6 -F \*

And also try to do adding probe with -v option again?

In my case, I could find the mangled symbol, but failed to setup the probe...

./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
Try to find probe point from debuginfo.
Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
Matched function: is_open
found inline addr: 0x8cad0
Probe point found: is_open+0
found inline addr: 0x8cac0
Probe point found: is_open+0
found inline addr: 0x8cab0
Probe point found: is_open+0
Matched function: is_open
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

Quoted 4 lines
[16568.176464] Failed to allocate trace_uprobe.(-22)
[16568.176468] Failed to parse address or file.

Aside of that mangled symbols with a @plt suffix do not work at all.

Oh, OK. I could reproduce it. I'll investigate it. :)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#4 lore raw
Quoted 39 lines · On Wed, 21 Sep 2016 12:37:59 +0900 Masami Hiramatsu <[email protected]> wrote:
Quoted 36 lines · On Tue, 20 Sep 2016 01:32:04 +0200 (CEST) Thomas Gleixner <[email protected]>…

Masami,

Quoted 7 lines · Masami Hiramatsu

Hmm, could you try to find mangled symbol from the library as below?

./perf probe -x /usr/lib64/libstdc++.so.6 -F \*

And also try to do adding probe with -v option again?

In my case, I could find the mangled symbol, but failed to setup the probe...

./perf probe -x /usr/lib/x86_64-linux-gnu/libstdc++.so.6 -v _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv probe-definition(0): _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv symbol:_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
symbol:catch file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:throw file:(null) line:0 offset:0 return:0 lazy:(null)
symbol:rethrow file:(null) line:0 offset:0 return:0 lazy:(null)
Open Debuginfo file: /usr/lib/debug/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
Try to find probe point from debuginfo.
Symbol _X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv address found : 8c620
Matched function: is_open
found inline addr: 0x8cad0
Probe point found: is_open+0
found inline addr: 0x8cac0
Probe point found: is_open+0
found inline addr: 0x8cab0
Probe point found: is_open+0
Matched function: is_open
An error occurred in debuginfo analysis (-2).
Trying to use symbols.
Opening /sys/kernel/tracing//uprobe_events write=1
Writing event: p:probe_libstdc++/_X_ZNKSt13basic_fstreamIcSt11char_traitsIcEE7is_openEv /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22:0x8c620 Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)

Quoted 7 lines
Quoted 4 lines
[16568.176464] Failed to allocate trace_uprobe.(-22)
[16568.176468] Failed to parse address or file.

Aside of that mangled symbols with a @plt suffix do not work at all.

Oh, OK. I could reproduce it. I'll investigate it. :)

Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.

1) No mangle/demangle symbol support. This needs a) fix option parser to accept C++ method name correctly, b) mangle that method, and c) demangle it when we show the probe point.

2) @plt symbol should be skipped since the real function will be called(jumped) from plt, when searching probe point.

3) Event group name is including some characters which can not be used (e.g. "++"). Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
#5 lore raw
Quoted 8 lines · Masami Hiramatsu

On Wed, 21 Sep 2016 12:37:59 +0900
Masami Hiramatsu <[email protected]> wrote:

Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.

1) No mangle/demangle symbol support. This needs a) fix option parser to accept C++ method name correctly, b) mangle that method, and c) demangle it when we show the probe point.

I think we should also allow using the mangled name, supporting (de)mangling is of course nice, but not required per-se. Esp. since there's more than just C++ mangling. Having the mangling stuff as optional makes it a convenience but still allows using this on other languages for which we do not support the mangling.

Quoted 5 lines

2) @plt symbol should be skipped since the real function will be called(jumped) from plt, when searching probe point.

3) Event group name is including some characters which can not be used (e.g. "++"). Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)

What is the problem with the '+' character? Why cannot this be used?

#6 lore raw
Quoted 10 lines · On Thu, 22 Sep 2016 10:54:06 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 8 lines · Masami Hiramatsu

On Wed, 21 Sep 2016 12:37:59 +0900
Masami Hiramatsu <[email protected]> wrote:

Hmm, I found there are 3 issues in perf-probe to define event on C++ libs.

1) No mangle/demangle symbol support. This needs a) fix option parser to accept C++ method name correctly, b) mangle that method, and c) demangle it when we show the probe point.

Quoted 5 lines

I think we should also allow using the mangled name, supporting (de)mangling is of course nice, but not required per-se. Esp. since there's more than just C++ mangling. Having the mangling stuff as optional makes it a convenience but still allows using this on other languages for which we do not support the mangling.

Agreed. I also learned that mangling depends on the compiler and no good way to get demangled name to mangled name. Moreover, the demangled name will include arguments, templates etc. because of supporting polymorphism. At this phase, I would like to support only "method name" or "mangled name".

Quoted 7 lines
Quoted 5 lines

2) @plt symbol should be skipped since the real function will be called(jumped) from plt, when searching probe point.

3) Event group name is including some characters which can not be used (e.g. "++"). Thus we have to remove it or replace it with '_' or other alphabet.("PP"?)

What is the problem with the '+' character? Why cannot this be used?

In the ftrace, it is not considered to use such characters for event/group name, because the name is used for generating symbols for tracepoints. IOW, other tracepoint events have the name same as C naming conventions. (Maybe that is also good for out-of-tree tools and scripts to parse it.)

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>