perf probe issues

DISCUSSION Arnaldo Carvalho de Melo 2 participants 4 messages Jun 14, 2016 – Jun 15, 2016

Share & tools

Arnaldo Carvalho de Melo

author

2016-06-14 17:05 UTC #0 lore raw

Hi Masami,

While investigating a problem in the annotation code reported by
Linus, with "kaslr" on the kernel command line and this in .config:

[root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config CONFIG_RANDOMIZE_BASE=y
# CONFIG_DEBUG_INFO is not set
[root@jouet ~]#

[root@jouet ~]# grep -w vfs_write /proc/kallsyms ffffffffac2404c0 T vfs_write
[root@jouet ~]# perf probe vfs_write
vfs_write is out of .text, skip it.
Error: Failed to add events.
[root@jouet ~]# perf probe -v vfs_write
probe-definition(0): vfs_write
symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
vfs_write is out of .text, skip it.
Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@jouet ~]#

Can you try to test with this config to investigate this problem?

We need to improve the error messages, for instance:

"Could not open debuginfo. Try to use symbols."

I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it doesn't manage to find a vmlinux file, it will try to use kallsyms as a fallback, is that what is happening here?

- Arnaldo

Masami Hiramatsu

2016-06-15 02:02 UTC #1 lore raw

Quoted 26 lines · On Tue, 14 Jun 2016 14:05:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Hi Masami,
While investigating a problem in the annotation code reported by
Linus, with "kaslr" on the kernel command line and this in .config:
[root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config CONFIG_RANDOMIZE_BASE=y
# CONFIG_DEBUG_INFO is not set
[root@jouet ~]#
[root@jouet ~]# grep -w vfs_write /proc/kallsyms ffffffffac2404c0 T vfs_write
[root@jouet ~]# perf probe vfs_write
vfs_write is out of .text, skip it.
Error: Failed to add events.
[root@jouet ~]# perf probe -v vfs_write
probe-definition(0): vfs_write
symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
vfs_write is out of .text, skip it.
Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@jouet ~]#

Quoted 1 line

Can you try to test with this config to investigate this problem?

Yes, I guess it missed to get start_text because of KASLR...

Quoted 8 lines

We need to improve the error messages, for instance:
"Could not open debuginfo. Try to use symbols."
I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it doesn't manage to find a vmlinux file, it will try to use kallsyms as a fallback, is that what is happening here?

Let me check that the perf symbol/map object uses kallsyms for kernel. Anyway, if the target is not the kernel, do we change the message as "trying to use symbols"?

Quoted 2 lines

- Arnaldo

Signature · 1 line

Masami Hiramatsu <[email protected]>

Arnaldo Carvalho de Melo

author

2016-06-15 17:43 UTC #2 lore raw

Em Wed, Jun 15, 2016 at 11:02:01AM +0900, Masami Hiramatsu escreveu:

Quoted 34 lines

Quoted 24 lines · On Tue, 14 Jun 2016 14:05:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
While investigating a problem in the annotation code reported by
Linus, with "kaslr" on the kernel command line and this in .config:
[root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config CONFIG_RANDOMIZE_BASE=y
# CONFIG_DEBUG_INFO is not set
[root@jouet ~]#
[root@jouet ~]# grep -w vfs_write /proc/kallsyms ffffffffac2404c0 T vfs_write
[root@jouet ~]# perf probe vfs_write
vfs_write is out of .text, skip it.
Error: Failed to add events.
[root@jouet ~]# perf probe -v vfs_write
probe-definition(0): vfs_write
symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
vfs_write is out of .text, skip it.
Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@jouet ~]#
Quoted 1 line
Can you try to test with this config to investigate this problem?
Yes, I guess it missed to get start_text because of KASLR...
Quoted 3 lines
We need to improve the error messages, for instance:
"Could not open debuginfo. Try to use symbols."

Quoted 7 lines

Quoted 3 lines
I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it doesn't manage to find a vmlinux file, it will try to use kallsyms as a fallback, is that what is happening here?
Let me check that the perf symbol/map object uses kallsyms for kernel. Anyway, if the target is not the kernel, do we change the message as "trying to use symbols"?

Well, it is _always_ trying to use "symbols" as in "information to map an address to a symbol", but this information can come from DWARF, kallsyms, ELF symbol tables, JIT /tmp/ maps, etc :-)

So, if it is going to use kallsyms:

"Trying to use kallsyms"

ELF symtol tables:

"Trying to use the ELF symbol table"

etc.

- Arnaldo

Masami Hiramatsu

2016-06-15 21:33 UTC #3 lore raw

Quoted 10 lines · On Wed, 15 Jun 2016 14:43:38 -0300 Arnaldo Carvalho de Melo <[email protected]>…

Em Wed, Jun 15, 2016 at 11:02:01AM +0900, Masami Hiramatsu escreveu:
Quoted 9 lines
Quoted 7 lines · On Tue, 14 Jun 2016 14:05:04 -0300 Arnaldo Carvalho de Melo <[email protected]>…
While investigating a problem in the annotation code reported by
Linus, with "kaslr" on the kernel command line and this in .config:
[root@jouet ~]# egrep 'CONFIG_(RANDOMIZE_BASE|DEBUG_INFO)' ~acme/git/build/v4.7.0-rc3/.config CONFIG_RANDOMIZE_BASE=y
# CONFIG_DEBUG_INFO is not set
[root@jouet ~]#

Hmm, could you give me more info? I tried same thing (on Ubuntu 16.04 kernel(4.4 based) which support kaslr by default), but perf probe succeeded to put an event on vfs_write (I ensured by <debugfs>/kprobes/list). What would you get if you grep _etext in /proc/kallsyms?
(Ah, perf probe must change error message if it failed to find _etext
from kallsyms...)

Quoted 37 lines

Quoted 25 lines
Quoted 17 lines
[root@jouet ~]# grep -w vfs_write /proc/kallsyms ffffffffac2404c0 T vfs_write
[root@jouet ~]# perf probe vfs_write
vfs_write is out of .text, skip it.
Error: Failed to add events.
[root@jouet ~]# perf probe -v vfs_write
probe-definition(0): vfs_write
symbol:vfs_write file:(null) line:0 offset:0 return:0 lazy:(null) 0 arguments
Looking at the vmlinux_path (8 entries long)
Using /lib/modules/4.7.0-rc3/build/vmlinux for symbols
Could not open debuginfo. Try to use symbols.
vfs_write is out of .text, skip it.
Error: Failed to add events. Reason: No such file or directory (Code: -2)
[root@jouet ~]#
Quoted 1 line
Can you try to test with this config to investigate this problem?
Yes, I guess it missed to get start_text because of KASLR...
Quoted 3 lines
We need to improve the error messages, for instance:
"Could not open debuginfo. Try to use symbols."
Quoted 7 lines
Quoted 3 lines
I think should be rewritten as: "Trying to use kallsyms", no? I.e. if it doesn't manage to find a vmlinux file, it will try to use kallsyms as a fallback, is that what is happening here?
Let me check that the perf symbol/map object uses kallsyms for kernel. Anyway, if the target is not the kernel, do we change the message as "trying to use symbols"?
Well, it is _always_ trying to use "symbols" as in "information to map an address to a symbol", but this information can come from DWARF, kallsyms, ELF symbol tables, JIT /tmp/ maps, etc :-)

Quoted 7 lines

So, if it is going to use kallsyms:
"Trying to use kallsyms"
ELF symtol tables:
"Trying to use the ELF symbol table"

OK, I'll do that.

Quoted 4 lines

etc.
- Arnaldo

Signature · 1 line

Masami Hiramatsu <[email protected]>