Skip to contentSkip to conversation

perf tools: Call graph from Intel BTS

DISCUSSION Adrian Hunter 3 participants 5 messages Jul 11, 2014 – Jul 12, 2014

lore
Share & tools
lore
#0 lore raw

Hi

Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and extended perf tools to enable the creation of context-sensitive call graphs. A screen shot from an example call graph from tracing "ls" is shown here:

http://git.infradead.org/~ahunter/call-graph-bts-ls.png

The patches are here:

http://git.infradead.org/users/ahunter/linux-perf.git

There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.

The perf tools python API has been extended to facilitate exporting to databases, a script is provided to export to PostgreSQL, and another script reads the database and displays the call graph.

The Intel BTS data provides branches (address from / to) but to create the call graph the type of instruction (call, return etc) is needed. That is found by reading the object code, but for the kernel that requires an accurate image. A copy of /proc/kcore is needed but it must be made under the same conditions as the trace. That has proven sufficiently awkward to warrant creating a script to do it.

Here is how the "ls" call graph was created:

$ sudo bash -c 'echo 0 > /proc/sys/kernel/kptr_restrict'
$ sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid'
$ sudo bash -c 'chmod go+rx /sys/kernel/debug'
$ ~/libexec/perf-core/perf-with-kcore record bts_ls  --per-thread -e intel_bts// -- ls
Recording
Using /home/ahunter/bin/perf
perf version 3.16.rc2.linux.local.gaabf
/home/ahunter/bin/perf record -o bts_ls/perf.data --per-thread -e intel_bts//  -- ls
CREDITS    drivers               ipc            MAINTAINERS   modules.order   REPORTING-BUGS  signing_key.priv  tools             vmlinux.o
arch       crypto                firmware       Kconfig       Makefile        Module.symvers  samples           signing_key.x509  usr             x509.genkey
block      fs                    include        kernel        mm              net             scripts           sound             virt
bts_ls     COPYING               Documentation  init          Kbuild          lib             modules.builtin   README            security        System.map        vmlinux
[ perf record: Woken up 100 times to write data ]
[ perf record: Captured and wrote 6.945 MB bts_ls/perf.data ]
Copying kcore
Done
$ ~/libexec/perf-core/perf-with-kcore script bts_ls -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls
Using /home/ahunter/bin/perf
perf version 3.16.rc2.linux.local.gaabf
/home/ahunter/bin/perf script -i bts_ls/perf.data --kallsyms=bts_ls/kcore_dir/kallsyms -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls
2014-07-11 15:13:38.313702 Creating database...
2014-07-11 15:13:39.150277 Writing to intermediate files...
2014-07-11 15:13:40.182816 Copying to database...
2014-07-11 15:13:41.776815 Removing intermediate files...
2014-07-11 15:13:41.783443 Adding primary keys
2014-07-11 15:13:44.290786 Adding foreign keys
2014-07-11 15:13:44.974865 Done
$ ~/libexec/perf-core/scripts/python/call-graph-from-postgresql.py bts_ls
$

Regards
Adrian

#1 lore raw

Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:

Quoted 3 lines

There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.

We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like:

commit 244c87b15b124914827f3ce28d8e70c8d147c9d0

Author: Adrian Hunter <[email protected]>
Date:   Wed Jun 11 09:33:17 2014 +0300

perf tools: Fix the value used for unknown pids

The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups.

Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence.

But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch:

[acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0
tools/perf/util/session.c: thread = machine__findnew_thread(&session->machines.host, 0, 0);
tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0);

tools/perf/tests/thread-mg-share.c:	t1     = machine__findnew_thread(machine, 0, 1);
tools/perf/tests/thread-mg-share.c:	t2     = machine__findnew_thread(machine, 0, 2);
tools/perf/tests/thread-mg-share.c:	t3     = machine__findnew_thread(machine, 0, 3);

[acme@ssdandy linux]$

So I think that one way to reduce the size of that branch is to do just that:
start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled.

You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat.

- Arnaldo

#2 lore raw
Quoted 24 lines · Arnaldo Carvalho de Melo

Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:

Quoted 3 lines

There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.

We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like:

commit 244c87b15b124914827f3ce28d8e70c8d147c9d0

Author: Adrian Hunter <[email protected]>
Date:   Wed Jun 11 09:33:17 2014 +0300

perf tools: Fix the value used for unknown pids

The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups.

Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence.

Quoted 8 lines

But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch:

[acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0
tools/perf/util/session.c: thread = machine__findnew_thread(&session->machines.host, 0, 0);

That's the idle thread.

Quoted 4 lines

tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0);

tools/perf/tests/thread-mg-share.c:	t1     = machine__findnew_thread(machine, 0, 1);
tools/perf/tests/thread-mg-share.c:	t2     = machine__findnew_thread(machine, 0, 2);
tools/perf/tests/thread-mg-share.c:	t3     = machine__findnew_thread(machine, 0, 3);

Those are valid pids for that test.

Quoted 8 lines

[acme@ssdandy linux]$

So I think that one way to reduce the size of that branch is to do just that:
start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled.

You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat.

Sounds good, thanks! It is currently based on tip/perf/core from a few days ago, so the current patches should be mostly ok. I will make a selection and check them again.

#3 lore raw
In reply to Adrian Hunter
Quoted 4 lines · Adrian Hunter

Hi

Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and

There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later.

#4 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Adrian Hunter

Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and

There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later.

Currently that driver does not allow kernel tracing. Last time I tried it with that restriction removed, it would lose data on larger workloads. Generally, any loss of data ruins the call graph.

For me it does not make sense to try to support it, since I am primarily trying to pave the way for the Intel PT driver.