perf tools: Call graph from Intel BTS
Hi
Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and extended perf tools to enable the creation of context-sensitive call graphs. A screen shot from an example call graph from tracing "ls" is shown here:
http://git.infradead.org/~ahunter/call-graph-bts-ls.png
The patches are here:
http://git.infradead.org/users/ahunter/linux-perf.git
There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.
The perf tools python API has been extended to facilitate exporting to databases, a script is provided to export to PostgreSQL, and another script reads the database and displays the call graph.
The Intel BTS data provides branches (address from / to) but to create the call graph the type of instruction (call, return etc) is needed. That is found by reading the object code, but for the kernel that requires an accurate image. A copy of /proc/kcore is needed but it must be made under the same conditions as the trace. That has proven sufficiently awkward to warrant creating a script to do it.
Here is how the "ls" call graph was created:
$ sudo bash -c 'echo 0 > /proc/sys/kernel/kptr_restrict' $ sudo bash -c 'echo -1 > /proc/sys/kernel/perf_event_paranoid' $ sudo bash -c 'chmod go+rx /sys/kernel/debug' $ ~/libexec/perf-core/perf-with-kcore record bts_ls --per-thread -e intel_bts// -- ls Recording Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf record -o bts_ls/perf.data --per-thread -e intel_bts// -- ls CREDITS drivers ipc MAINTAINERS modules.order REPORTING-BUGS signing_key.priv tools vmlinux.o arch crypto firmware Kconfig Makefile Module.symvers samples signing_key.x509 usr x509.genkey block fs include kernel mm net scripts sound virt bts_ls COPYING Documentation init Kbuild lib modules.builtin README security System.map vmlinux [ perf record: Woken up 100 times to write data ] [ perf record: Captured and wrote 6.945 MB bts_ls/perf.data ] Copying kcore Done $ ~/libexec/perf-core/perf-with-kcore script bts_ls -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls Using /home/ahunter/bin/perf perf version 3.16.rc2.linux.local.gaabf /home/ahunter/bin/perf script -i bts_ls/perf.data --kallsyms=bts_ls/kcore_dir/kallsyms -s ~/libexec/perf-core/scripts/python/export-to-postgresql.py bts_ls branches calls 2014-07-11 15:13:38.313702 Creating database... 2014-07-11 15:13:39.150277 Writing to intermediate files... 2014-07-11 15:13:40.182816 Copying to database... 2014-07-11 15:13:41.776815 Removing intermediate files... 2014-07-11 15:13:41.783443 Adding primary keys 2014-07-11 15:13:44.290786 Adding foreign keys 2014-07-11 15:13:44.974865 Done $ ~/libexec/perf-core/scripts/python/call-graph-from-postgresql.py bts_ls $
Regards
Adrian
Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:
Quoted 3 lines
There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.
We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like:
commit 244c87b15b124914827f3ce28d8e70c8d147c9d0
Author: Adrian Hunter <[email protected]> Date: Wed Jun 11 09:33:17 2014 +0300
perf tools: Fix the value used for unknown pids
The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups.
Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence.
But then one by one they need to be reviewed to check if the changes were made to the whole tools/perf/ tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch:
[acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep machine__findnew_thread | grep 0tools/perf/util/session.c: thread = machine__findnew_thread(&session->machines.host, 0, 0);tools/perf/tests/thread-mg-share.c: leader = machine__findnew_thread(machine, 0, 0);
tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3);
[acme@ssdandy linux]$
So I think that one way to reduce the size of that branch is to do just that:
start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole current tools/perf/ tree, then ask for this patch to be pulled.
You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat.
- Arnaldo
Quoted 4 lines · Adrian Hunter
Hi
Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and
There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later.
Quoted 24 lines · Arnaldo Carvalho de Melo
Em Fri, Jul 11, 2014 at 05:36:41PM +0300, Adrian Hunter escreveu:
Quoted 3 lines
There are many perf tools patches and it would be helpful to start considering how to get them into mainline. Many need to wait for the driver, but others could be taken sooner.
We can go on looking at each of the patches to see which ones can be cherry picked, i.e. the ones that are fixes and not related to the work you're doing, like:
commit
244c87b15b124914827f3ce28d8e70c8d147c9d0Author: Adrian Hunter <[email protected]> Date: Wed Jun 11 09:33:17 2014 +0300perf tools: Fix the value used for unknown pids
The value used for unknown pids cannot be zero because that is used by the "idle" task. Use -1 instead. Also handle the unknown pid case when creating map groups.
Note that, threads with an unknown pid should not occur because fork (or synthesized) events precede the thread's existence.
Quoted 8 lines
Signed-off-by Adrian Hunter <[email protected]>But then one by one they need to be reviewed to check if the changes were made to the whole
tools/perf/tree and if perhaps something new came along since you changed some assumption, like 0 meaning unknown thread, in the above patch:[acme@ssdandy linux]$ find tools -name "*.[ch]" | xargs grep
machine__findnew_thread| grep 0tools/perf/util/session.c: thread =machine__findnew_thread(&session->machines.host, 0, 0);
That's the idle thread.
Quoted 4 lines
tools/perf/tests/thread-mg-share.c: leader =machine__findnew_thread(machine, 0, 0);tools/perf/tests/thread-mg-share.c: t1 = machine__findnew_thread(machine, 0, 1); tools/perf/tests/thread-mg-share.c: t2 = machine__findnew_thread(machine, 0, 2); tools/perf/tests/thread-mg-share.c: t3 = machine__findnew_thread(machine, 0, 3);
Those are valid pids for that test.
Quoted 8 lines
[acme@ssdandy linux]$
So I think that one way to reduce the size of that branch is to do just that:
start fresh from tip/perf/core, and go cherry picking those patches, making sure that they take into account the whole currenttools/perf/tree, then ask for this patch to be pulled.You could then rebase the old branch on top of the resulting branch once it is merged upstream, rinse repeat.
Sounds good, thanks! It is currently based on tip/perf/core from a few days ago, so the current patches should be mostly ok. I will make a selection and check them again.
Quoted 7 lines · Peter Zijlstra
Quoted 2 lines · Adrian Hunter
Alexander Shishkin is working on the Intel PT driver for perf and has included a driver for Intel BTS. I have taken that and
There is already a BTS driver, although I've not used it ever, since there's no useful tool for it. One way would be to use that, and migrate to PT data later.
Currently that driver does not allow kernel tracing. Last time I tried it with that restriction removed, it would lose data on larger workloads. Generally, any loss of data ruins the call graph.
For me it does not make sense to try to support it, since I am primarily trying to pave the way for the Intel PT driver.