Skip to contentSkip to conversation

perf cannot see call graph, visible in gdb

DISCUSSION Mark Hills 4 participants 5 messages Sep 5, 2013 – Sep 9, 2013

lore
Share & tools
lore

Mark Hills

author
#0 lore raw

I have a pre-compiled .so library, which was given to me for profiling.

gdb can see a complete call stack -- both the library and my own code, as expected.

But in perf the callgraph for the library is not present. I'm using "perf record -g", and the callgraph for the other code is seen.

In what cases could gdb see the stack, but perf cannot?

Many thanks

Signature · 49 lines
Mark


---
# System is Scientific Linux 6.3 (derived from RedHat 6.3)

$ uname -a
Linux star 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 20:37:17 CST 2013 x86_64 x86_64 x86_64 GNU/Linux

$ rpm -q perf
perf-2.6.32-358.el6.x86_64

$ rpm -q gdb
gdb-7.2-56.el6.x86_64

# The library which is missing the callgraph is librj.so

$ perf script -L
[...]
radford. 16009 1117798.447638: cycles: 
            7ff2836dd906 RjMetaDataGetStr (/mnt/scratch/radford/librj.so)

radford. 16009 1117798.471449: cycles: 
            7ff2837213a2 RjFile::getChar() (/mnt/scratch/radford/librj.so)

radford. 16102 1117798.707682: cycles: 
        ffffffffa0fa9281 rpc_task_release_client (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa0fb1f7b rpc_release_resources_task (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa0fb2a84 __rpc_execute (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa0fb2cc1 rpc_execute (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa0fa93a5 rpc_run_task (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa0fa94c2 rpc_call_sync (/lib/modules/2.6.32-358.el6.x86_64/kernel/net/sunrpc/sunrpc.ko)
        ffffffffa106af6d nfs3_rpc_wrapper.clone.0 (/lib/modules/2.6.32-358.el6.x86_64/kernel/fs/nfs/nfs.ko)
        ffffffffa106b34c nfs3_proc_access (/lib/modules/2.6.32-358.el6.x86_64/kernel/fs/nfs/nfs.ko)
        ffffffffa10531e9 nfs_do_access (/lib/modules/2.6.32-358.el6.x86_64/kernel/fs/nfs/nfs.ko)
        ffffffffa10534b8 nfs_permission (/lib/modules/2.6.32-358.el6.x86_64/kernel/fs/nfs/nfs.ko)
        ffffffff8119053d __link_path_walk ([kernel.kallsyms])
        ffffffff811909ca __link_path_walk ([kernel.kallsyms])
        ffffffff8119174a path_walk ([kernel.kallsyms])
        ffffffff8119191b do_path_lookup ([kernel.kallsyms])
        ffffffff811925a7 user_path_at ([kernel.kallsyms])
        ffffffff811869bc vfs_fstatat ([kernel.kallsyms])
        ffffffff81186b2b vfs_stat ([kernel.kallsyms])
        ffffffff81186b54 sys_newstat ([kernel.kallsyms])
        ffffffff8100b072 system_call_fastpath ([kernel.kallsyms])
              320a6da735 __xstat64 (/lib64/libc-2.12.so)
            7ff282d09247 Evaluate(RjNode*, RjHiderGlobals*) (/home/mark/built/rathiders.so)
            7ff283701585 RjHiderEvaluate (/mnt/scratch/radford/librj.so)
            7ff283701585 RjHiderEvaluate (/mnt/scratch/radford/librj.so)
#1 lore raw
Quoted 9 lines · Mark Hills

I have a pre-compiled .so library, which was given to me for profiling.

gdb can see a complete call stack -- both the library and my own code, as expected.

But in perf the callgraph for the library is not present. I'm using "perf record -g", and the callgraph for the other code is seen.

In what cases could gdb see the stack, but perf cannot?

No frame pointer.

Recompile with -fno-omit-frame-pointer

Or if you have a new enough perf, you can use -g dwarf to enable dwarf backtracing, but it's very slow and also doesn't handle all situations gdb handles.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
author
#2 lore raw
Quoted 12 lines · Andi Kleen
Quoted 9 lines · Mark Hills

I have a pre-compiled .so library, which was given to me for profiling.

gdb can see a complete call stack -- both the library and my own code, as expected.

But in perf the callgraph for the library is not present. I'm using "perf record -g", and the callgraph for the other code is seen.

In what cases could gdb see the stack, but perf cannot?

Quoted 7 lines

No frame pointer.

Recompile with -fno-omit-frame-pointer

Or if you have a new enough perf, you can use -g dwarf to enable dwarf backtracing, but it's very slow and also doesn't handle all situations gdb handles.

Thank you, much appreciated.

A newer perf appears to need a newer kernel too, and it's not practical for me to break away from the RedHat kernel at the moment (currently on 2.6.32-358.14.1.el6)

Also I looked to cherry-pick the relevant patches, but it seems this is non-trivial.

I'll speak to the author of the library in question but may have to re-visit the dwarf behaviour; omit-frame-pointer may have been used with performance in mind.

Thanks

Signature · 1 line
Mark
#3 lore raw

Adding Jiri.

Quoted 22 lines · Mark Hills
Quoted 12 lines · Andi Kleen
Quoted 9 lines · Mark Hills

I have a pre-compiled .so library, which was given to me for profiling.

gdb can see a complete call stack -- both the library and my own code, as expected.

But in perf the callgraph for the library is not present. I'm using "perf record -g", and the callgraph for the other code is seen.

In what cases could gdb see the stack, but perf cannot?

Quoted 7 lines

No frame pointer.

Recompile with -fno-omit-frame-pointer

Or if you have a new enough perf, you can use -g dwarf to enable dwarf backtracing, but it's very slow and also doesn't handle all situations gdb handles.

Quoted 5 lines

Thank you, much appreciated.

A newer perf appears to need a newer kernel too, and it's not practical for me to break away from the RedHat kernel at the moment (currently on 2.6.32-358.14.1.el6)

Any updates to RHEL6 for dwarf callchains?

David

Quoted 10 lines

Also I looked to cherry-pick the relevant patches, but it seems this is non-trivial.

I'll speak to the author of the library in question but may have to re-visit the dwarf behaviour; omit-frame-pointer may have been used with performance in mind.

Thanks

#4 lore raw
Quoted 25 lines · David Ahern

Adding Jiri.

Quoted 22 lines · Mark Hills
Quoted 19 lines · Andi Kleen
Quoted 9 lines · Mark Hills

I have a pre-compiled .so library, which was given to me for profiling.

gdb can see a complete call stack -- both the library and my own code, as expected.

But in perf the callgraph for the library is not present. I'm using "perf record -g", and the callgraph for the other code is seen.

In what cases could gdb see the stack, but perf cannot?

No frame pointer.

Recompile with -fno-omit-frame-pointer

Or if you have a new enough perf, you can use -g dwarf to enable dwarf backtracing, but it's very slow and also doesn't handle all situations gdb handles.

Quoted 8 lines
Quoted 5 lines

Thank you, much appreciated.

A newer perf appears to need a newer kernel too, and it's not practical for me to break away from the RedHat kernel at the moment (currently on 2.6.32-358.14.1.el6)

Any updates to RHEL6 for dwarf callchains?

unfortunately no news in RHEL6 for this one

jirka