perf: Add 'perf kmem' tool

This tool is mostly a perf version of kmemtrace-user.

The following information is provided by this tool:

- the total amount of memory allocated and fragmentation per call-site
- the total amount of memory allocated and fragmentation per allocation
- total memory allocated and fragmentation in the collected dataset
- ...

 # ./perf kmem record
 ^C
 # ./perf kmem --stat caller --stat alloc -l 10

------------------------------------------------------------------------------
Callsite | Total_alloc/Per | Total_req/Per | Hit | Fragmentation
------------------------------------------------------------------------------

 0xc052f37a        |   790528/4096   |   790528/4096   |    193 |    0.000%
 0xc0541d70        |   524288/4096   |   524288/4096   |    128 |    0.000%
 0xc051cc68        |   481600/200    |   481600/200    |   2408 |    0.000%
 0xc0572623        |   297444/676    |   297440/676    |    440 |    0.001%
 0xc05399f1        |    73476/164    |    73472/164    |    448 |    0.005%
 0xc05243bf        |    51456/256    |    51456/256    |    201 |    0.000%
 0xc0730d0e        |    31844/497    |    31808/497    |     64 |    0.113%
 0xc0734c4e        |    17152/256    |    17152/256    |     67 |    0.000%
 0xc0541a6d        |    16384/128    |    16384/128    |    128 |    0.000%
 0xc059c217        |    13120/40     |    13120/40     |    328 |    0.000%
 0xc0501ee6        |    11264/88     |    11264/88     |    128 |    0.000%
 0xc04daef0        |     7504/682    |     7128/648    |     11 |    5.011%
 0xc04e14a3        |     4216/191    |     4216/191    |     22 |    0.000%
 0xc05041ca        |     3524/44     |     3520/44     |     80 |    0.114%
 0xc0734fa3        |     2104/701    |     1620/540    |      3 |   23.004%
 0xc05ec9f1        |     2024/289    |     2016/288    |      7 |    0.395%
 0xc06a1999        |     1792/256    |     1792/256    |      7 |    0.000%
 0xc0463b9a        |     1584/144    |     1584/144    |     11 |    0.000%
 0xc0541eb0        |     1024/16     |     1024/16     |     64 |    0.000%
 0xc06a19ac        |      896/128    |      896/128    |      7 |    0.000%
 0xc05721c0        |      772/12     |      768/12     |     64 |    0.518%
 0xc054d1e6        |      288/57     |      280/56     |      5 |    2.778%
 0xc04b562e        |      157/31     |      154/30     |      5 |    1.911%
 0xc04b536f        |       80/16     |       80/16     |      5 |    0.000%
 0xc05855a0        |       64/64     |       36/36     |      1 |   43.750%

------------------------------------------------------------------------------

------------------------------------------------------------------------------
Alloc Ptr | Total_alloc/Per | Total_req/Per | Hit | Fragmentation
------------------------------------------------------------------------------

 0xda884000        |  1052672/4096   |  1052672/4096   |    257 |    0.000%
 0xda886000        |   262144/4096   |   262144/4096   |     64 |    0.000%
 0xf60c7c00        |    16512/128    |    16512/128    |    129 |    0.000%
 0xf59a4118        |    13120/40     |    13120/40     |    328 |    0.000%
 0xdfd4b2c0        |    11264/88     |    11264/88     |    128 |    0.000%
 0xf5274600        |     7680/256    |     7680/256    |     30 |    0.000%
 0xe8395000        |     5948/594    |     5464/546    |     10 |    8.137%
 0xe59c3c00        |     5748/479    |     5712/476    |     12 |    0.626%
 0xf4cd1a80        |     3524/44     |     3520/44     |     80 |    0.114%
 0xe5bd1600        |     2892/482    |     2856/476    |      6 |    1.245%
 ...               | ...             | ...             | ...    | ...

------------------------------------------------------------------------------

SUMMARY
=======
Total bytes requested: 2333626
Total bytes allocated: 2353712
Total bytes wasted on internal fragmentation: 20086 Internal fragmentation: 0.853375%

TODO:
- show sym+offset in 'callsite' column

- show cross node allocation stats

- collect more useful stats?
- ...

Pekka, Eduard and the other slab hackers might have ideas about what other stats they generally like to see to judge the health of a workload (or system).

If this iteration looks good to the slab folks then i can apply it as-is and we can do the other changes relative to that. It looks good to me as a first step, and it's functional already.

The kmem trace events can replace the functions of kmemtrace tracer.

And kmemtrace-user can be modified to use trace events. (But after cloning the git repo, I found it's still based on the original relay version..), not to mention now we have 'perf kmem' tool.

Signed-off-by Li Zefan <[email protected]>

Quoted 2 lines

TODO:
- show sym+offset in 'callsite' column

The way to print symbolic information for the 'callsite' column is to fill in and walk the thread->DSO->symbol trees that all perf tools maintain:

	/* simplified, without error handling */

	ip = event->ip.ip;

	thread = threads__findnew(event->ip.pid);

	map = thread__find_map(thread, ip);

	ip = map->map_ip(map, ip); /* map absolute RIP into DSO-relative one */

	sym = map__find_symbol(map, ip, symbol_filter);

then sym->name is the string that can be printed out. This works in a symmetric way for both kernel-space and user-space symbols. (Call-chain information can be captured and displayed too.)

( 'Alloc Ptr' symbolization is harder, but it would be useful too i
think, to map it back to the slab cache name. )

Quoted 1 line

- show cross node allocation stats

I checked and we appear to have all the right events for that - the node ID is being traced consistently AFAICS.

Quoted 2 lines

- collect more useful stats?
- ...

Pekka, Eduard and the other slab hackers might have ideas about what other stats they generally like to see to judge the health of a workload (or system).

If this iteration looks good to the slab folks then i can apply it as-is and we can do the other changes relative to that. It looks good to me as a first step, and it's functional already.

Li Zefan kirjoitti:

Quoted 8 lines

The kmem trace events can replace the functions of kmemtrace tracer.

And kmemtrace-user can be modified to use trace events. (But after cloning the git repo, I found it's still based on the original relay version..), not to mention now we have 'perf kmem' tool.

Quoted 1 line

Signed-off-by Li Zefan <[email protected]>

NAK for the time being. "perf kmem" output is not yet as good as that of kmemtrace-user.

From the git repo:

Quoted 9 lines · Pekka Enberg

Li Zefan kirjoitti:

Quoted 8 lines

The kmem trace events can replace the functions of kmemtrace tracer.

And kmemtrace-user can be modified to use trace events. (But after cloning the git repo, I found it's still based on the original relay version..), not to mention now we have 'perf kmem' tool.

Quoted 5 lines

Quoted 1 line

Signed-off-by Li Zefan <[email protected]>

NAK for the time being. "perf kmem" output is not yet as good as that of kmemtrace-user.

But is the current kmemtrace-user based on kmemtrace?

From the git repo:
http://repo.or.cz/w/kmemtrace-user.git

I found it's still based on relay.

Quoted 3 lines · Ingo Molnar

Pekka, Eduard and the other slab hackers might have ideas about what other stats they generally like to see to judge the health of a workload (or system).

kmalloc()/kfree() CPU ping-pong call-sites (i.e. alloc and free happening on different CPUs) is one interesting metric we haven't implemented yet. Valgrind massif tool type of output graph would be helpful as well:

http://valgrind.org/docs/manual/ms-manual.html

On Fri, Nov 20, 2009 at 10:14 AM, Ingo Molnar <[email protected]> wrote:

Quoted 3 lines

If this iteration looks good to the slab folks then i can apply it as-is and we can do the other changes relative to that. It looks good to me as a first step, and it's functional already.

Yeah, looks OK to me as the first step. Patch 2 looks premature, though, looking at the output of "perf kmem" from patch 1.

Acked-by Pekka Enberg <[email protected]>

Quoted 14 lines · Li Zefan

Quoted 13 lines · Pekka Enberg

Li Zefan kirjoitti:

Quoted 9 lines

The kmem trace events can replace the functions of kmemtrace tracer.

And kmemtrace-user can be modified to use trace events. (But after cloning the git repo, I found it's still based on the original relay version..), not to mention now we have 'perf kmem' tool.

Signed-off-by Li Zefan <[email protected]>

NAK for the time being. "perf kmem" output is not yet as good as that of kmemtrace-user.

Quoted 6 lines

But is the current kmemtrace-user based on kmemtrace?

From the git repo:
http://repo.or.cz/w/kmemtrace-user.git

I found it's still based on relay.

The "ftrace-temp" branch seems to have the ftrace based version in it. Eduard?

Regarding patch 2 - can we set some definitive benchmark threshold for that? I.e. a list of must-have features in 'perf kmem' before we can do it? 100% information and analysis equivalency with kmemtrace-user tool?

Ingo Molnar kirjoitti:

Quoted 4 lines

Regarding patch 2 - can we set some definitive benchmark threshold for that? I.e. a list of must-have features in 'perf kmem' before we can do it? 100% information and analysis equivalency with kmemtrace-user tool?

I'd be interested to hear Eduard's comment on that.

That said, I'll try to find some time to test "perf kmem" and provide feedback on that. I can ACK the patch when I'm happy with the output. :-)

I'm mostly interested in two scenarios: (1) getting a nice report on worst fragmented call-sites (perf kmem needs symbol lookup) and

(2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

---
 tools/perf/Makefile       |    1 +
 tools/perf/builtin-kmem.c |  578 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h      |    1 +
 tools/perf/perf.c         |   27 +-
 4 files changed, 594 insertions(+), 13 deletions(-)
 create mode 100644 tools/perf/builtin-kmem.c

Ingo Molnar kirjoitti:

Quoted 4 lines

Regarding patch 2 - can we set some definitive benchmark threshold for that? I.e. a list of must-have features in 'perf kmem' before we can do it? 100% information and analysis equivalency with kmemtrace-user tool?

I'd be interested to hear Eduard's comment on that.

That said, I'll try to find some time to test "perf kmem" and provide feedback on that. I can ACK the patch when I'm happy with the output. :-)

I'm mostly interested in two scenarios: (1) getting a nice report on worst fragmented call-sites (perf kmem needs symbol lookup) and (2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

Quoted 4 lines

(2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

Everything needed and machine-specific will be recorded in perf.data, so this should already been supported. I'll try it.

Quoted 7 lines · Li Zefan

---
 tools/perf/Makefile       |    1 +
 tools/perf/builtin-kmem.c |  578 +++++++++++++++++++++++++++++++++++++++++++++
 tools/perf/builtin.h      |    1 +
 tools/perf/perf.c         |   27 +-
 4 files changed, 594 insertions(+), 13 deletions(-)
 create mode 100644 tools/perf/builtin-kmem.c

btw., you might want to add it to command-list.txt as well (in a future patch), so that 'kmem' shows up in the default 'perf' output.

Plus a Documentation/perf-kmem.txt file will make sure that 'perf help kmem' and 'perf kmem --help' displays a help page, etc.

Quoted 4 lines · Li Zefan

Quoted 4 lines

(2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

Quoted 2 lines

Everything needed and machine-specific will be recorded in perf.data, so this should already been supported. I'll try it.

Right now the DSOs are not recorded in the perf.data - but it would be useful to add it and to turn perf.data into a self-sufficient capture of all relevant data, which can be analyzed on any box.

But ... even without that, perf is really fast and is supposed to build fine even in minimal (embedded) environments, so you can run it on the embedded board too. That's useful to get live inspection features like 'perf top', 'perf stat' and 'perf probe' anyway.

Hi Ingo,

Quoted 4 lines · Ingo Molnar

But ... even without that, perf is really fast and is supposed to build fine even in minimal (embedded) environments, so you can run it on the embedded board too. That's useful to get live inspection features like 'perf top', 'perf stat' and 'perf probe' anyway.

Maybe I'm just too damn lazy but if I don't go through the trouble of building my kernel on the box, I sure don't want to do that for perf either. [...]

Quoted 8 lines

Hi Ingo,

Quoted 4 lines · Ingo Molnar

But ... even without that, perf is really fast and is supposed to build fine even in minimal (embedded) environments, so you can run it on the embedded board too. That's useful to get live inspection features like 'perf top', 'perf stat' and 'perf probe' anyway.

Quoted 3 lines

Maybe I'm just too damn lazy but if I don't go through the trouble of building my kernel on the box, I sure don't want to do that for perf either. [...]

Well you'll need 'perf' on that box anyway, to be able to do 'perf kmem record'.

Ingo Molnar kirjoitti:

Quoted 13 lines · Pekka Enberg

Quoted 11 lines

Hi Ingo,

Quoted 4 lines · Ingo Molnar

But ... even without that, perf is really fast and is supposed to build fine even in minimal (embedded) environments, so you can run it on the embedded board too. That's useful to get live inspection features like 'perf top', 'perf stat' and 'perf probe' anyway.

Maybe I'm just too damn lazy but if I don't go through the trouble of building my kernel on the box, I sure don't want to do that for perf either. [...]

Quoted 2 lines

Well you'll need 'perf' on that box anyway, to be able to do 'perf kmem record'.

/me turns brains on

You're right, of course. With kmemtrace-user, I just copied the raw trace file from /sys/kernel. I wonder if that's a good enough reason to keep kmemtrace bits around?

Quoted 6 lines

Quoted 5 lines

(2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

Quoted 2 lines

Everything needed and machine-specific will be recorded in perf.data, so this should already been supported. I'll try it.

Right now the DSOs are not recorded in the perf.data - but it would be useful to add it and to turn perf.data into a self-sufficient capture of all relevant data, which can be analyzed on any box.

Em Fri, Nov 20, 2009 at 10:03:53AM +0100, Ingo Molnar escreveu:

Quoted 8 lines · Li Zefan

Quoted 6 lines

Quoted 5 lines

(2) doing "perf kmem record" on machine A (think embedded here) and then "perf kmem report" on machine B. I haven't tried kmemtrace-user for a while but it did support both of them quite nicely at some point.

Quoted 6 lines

Quoted 2 lines

Everything needed and machine-specific will be recorded in perf.data, so this should already been supported. I'll try it.

Right now the DSOs are not recorded in the perf.data - but it would be useful to add it and to turn perf.data into a self-sufficient capture of all relevant data, which can be analyzed on any box.

Well, the DSOs are recorded in perf.data, just not its symtabs, but now we have buildids, so we can ask for them to be installed on the other machine and it'll all work. Or should. :)

For instance:

[root@doppio linux-2.6-tip]# perf buildid-list -i perf.data | egrep 'vmlinux|nfs|libc-' ec8dd400904ddfcac8b1c343263a790f977159dc /lib64/libc-2.10.1.so 0da49504693a200583fda6f1b949e6d2f799e692 /usr/lib64/libnfsidmap_nsswitch.so.0.0.0 c90269c87eaf08559012a9fa29f60780743360cd /usr/lib64/libnfsidmap.so.0.3.0 18e7cc53db62a7d35e9d6f6c9ddc23017d38ee9a vmlinux 3982866276471cde6ac5821fdced42a7b1bfd848 [nfs] 1489007276a50005753e730198fd93dd05b2082f [nfsd] 5a128f082fe7fdcab6fb5d1b71935accb1f34383 [nfs_acl]
[root@doppio linux-2.6-tip]#

Now if I ask that the buildid for /usr/lib64/libnfsidmap.so.0.3.0 above to be installed, like this:

[root@doppio linux-2.6-tip]# yum install /usr/lib/debug/.build-id/c9/0269c87eaf08559012a9fa29f60780743360cd Loaded plugins: auto-update-debuginfo, refresh-packagekit
Found 44 installed debuginfo package(s)
Enabling fedora-debuginfo: Fedora 11 - x86_64 - Debug
Reading repository metadata in from local files
Enabling updates-debuginfo: Fedora 11 - x86_64 - Updates - Debug
Reading repository metadata in from local files
Setting up Install Process
Importing additional filelist information
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils-lib-debuginfo.x86_64 0:1.1.4-6.fc11 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================
Package Arch Version Repository Size
========================================================================
Installing:
nfs-utils-lib-debuginfo x86_64 1.1.4-6.fc11 fedora-debuginfo 174 k

Transaction Summary
========================================================================
Install 1 Package(s)
Upgrade 0 Package(s)

Total download size: 174 k
Is this ok [y/N]:

So now we have:

1) 'perf record' records the build-ids into perf.data
2) 'perf buildid-list' list them, distro specific porcelain needed to do the equivalent to the yum install above.
3) 'perf report' will only use the symtab in a file that has the matching build-id, if a build-id is found in the perf.data header for a particular DSO.

So we have a mechanism that is already present in several distros (build-id), that is in the kernel build process since ~2.6.23, and that avoids using mismatching DSOs when resolving symbols.

Quoted 3 lines

So we have a mechanism that is already present in several distros (build-id), that is in the kernel build process since ~2.6.23, and that avoids using mismatching DSOs when resolving symbols.

But what do we do if we have another box that runs say on a MIPS CPU, uses some minimal distro - and copy that perf.data over to an x86 box.

The idea is there to be some new mode of perf.data where all the relevant DSO contents (symtabs but also sections with instructions for perf annotate to work) are copied into perf.data, during or after data capture - on the box that does the recording.

Once we have everything embedded in the perf.data, analysis passes only have to work based on that particular perf.data - no external data.

Em Fri, Nov 20, 2009 at 05:41:10PM +0100, Ingo Molnar escreveu:

Quoted 6 lines

Quoted 3 lines

So we have a mechanism that is already present in several distros (build-id), that is in the kernel build process since ~2.6.23, and that avoids using mismatching DSOs when resolving symbols.

But what do we do if we have another box that runs say on a MIPS CPU, uses some minimal distro - and copy that perf.data over to an x86 box.

There would be no problem, it would be just a matter of installing the right -debuginfo packages, for MIPS.

Or the original, unstripped FS image sent to the machine with the MIPS cpu, if there aren't -debuginfo packages.

Either one, the right DSOs would be found by the buildids.

There are other scenarios, like a binary that gets updated while a long running perf record session runs, the way to differentiate between the two DSOs wouldn't be the name, but the buildid.

Quoted 5 lines

The idea is there to be some new mode of perf.data where all the relevant DSO contents (symtabs but also sections with instructions for perf annotate to work) are copied into perf.data, during or after data capture - on the box that does the recording.

Quoted 2 lines

Once we have everything embedded in the perf.data, analysis passes only have to work based on that particular perf.data - no external data.

Well, we can that, additionally, but think about stripped binaries, we would lose potentially a lot because the symtabs on that small machine would have poorer symtabs than the ones in an unstriped binary (or in a -debuginfo package).

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Fri, Nov 20, 2009 at 05:41:10PM +0100, Ingo Molnar escreveu:

Quoted 6 lines

Quoted 3 lines

So we have a mechanism that is already present in several distros (build-id), that is in the kernel build process since ~2.6.23, and that avoids using mismatching DSOs when resolving symbols.

But what do we do if we have another box that runs say on a MIPS CPU, uses some minimal distro - and copy that perf.data over to an x86 box.

Quoted 2 lines

There would be no problem, it would be just a matter of installing the right -debuginfo packages, for MIPS.

I havent tried this - is this really possible to do on an x86 box, with a typical distro? Can i install say Fedora PowerPC debuginfo packages on an x86 box, while also having the x86 debuginfo packages there?

Quoted 4 lines

I havent tried this - is this really possible to do on an x86 box, with a typical distro? Can i install say Fedora PowerPC debuginfo packages on an x86 box, while also having the x86 debuginfo packages there?

The best option would be to allow to specify a chroot parameter, where we can specify the embedded root filesystem on out machine.

I'm not even sure embedded distros even have this separate debug package crazyness, you simply build the distro with or without debuginfo.

Quoted 9 lines

Em Fri, Nov 20, 2009 at 05:41:10PM +0100, Ingo Molnar escreveu:

Quoted 6 lines

Quoted 3 lines

So we have a mechanism that is already present in several distros (build-id), that is in the kernel build process since ~2.6.23, and that avoids using mismatching DSOs when resolving symbols.

But what do we do if we have another box that runs say on a MIPS CPU, uses some minimal distro - and copy that perf.data over to an x86 box.

Quoted 2 lines

There would be no problem, it would be just a matter of installing the right -debuginfo packages, for MIPS.

I havent tried this - is this really possible to do on an x86 box, with a typical distro? Can i install say Fedora PowerPC debuginfo packages on an x86 box, while also having the x86 debuginfo packages there?

Quoted 14 lines

Or the original, unstripped FS image sent to the machine with the MIPS cpu, if there aren't -debuginfo packages.

Either one, the right DSOs would be found by the buildids.

There are other scenarios, like a binary that gets updated while a long running perf record session runs, the way to differentiate between the two DSOs wouldn't be the name, but the buildid.

Quoted 5 lines

The idea is there to be some new mode of perf.data where all the relevant DSO contents (symtabs but also sections with instructions for perf annotate to work) are copied into perf.data, during or after data capture - on the box that does the recording.

Quoted 7 lines

Quoted 2 lines

Once we have everything embedded in the perf.data, analysis passes only have to work based on that particular perf.data - no external data.

Well, we can that, additionally, but think about stripped binaries, we would lose potentially a lot because the symtabs on that small machine would have poorer symtabs than the ones in an unstriped binary (or in a -debuginfo package).

We should definitely use the widest and best quality information we can
- if it's available.

So even if we 'inline' any information from the box, if there's better info available at the time of analysis, we should use that too.

Basically what matters is the principle of 'what is possible'.

If a user records on a box and analyses on a different box, and we end up not doing something (and printing an error or displaying an empty profile) that could reasonably have been done, then the user will be unhappy and we might lose that user.

The user wont be unhappy about us using a big set of data sources that we can recover information from transparently. The user will be unhappy if we insist on (and force) a certain form of information source - such as debuginfo.

Quoted 3 lines

I havent tried this - is this really possible to do on an x86 box, with a typical distro? Can i install say Fedora PowerPC debuginfo packages on an x86 box, while also having the x86 debuginfo packages there?

The best option would be to allow to specify a chroot parameter, where we can specify the embedded root filesystem on out machine.

I'm not even sure embedded distros even have this separate debug package crazyness, you simply build the distro with or without debuginfo.

Quoted 4 lines

You're right, of course. With kmemtrace-user, I just copied the raw trace file from /sys/kernel. I wonder if that's a good enough reason to keep kmemtrace bits around?

Not really. If then a light-weight recording app could be made but i'd rather wait for actual usecases to pop up.

Quoted 7 lines · Ingo Molnar

Quoted 4 lines

You're right, of course. With kmemtrace-user, I just copied the raw trace file from /sys/kernel. I wonder if that's a good enough reason to keep kmemtrace bits around?

Not really. If then a light-weight recording app could be made but i'd rather wait for actual usecases to pop up.

Hmm, but isn't this an actual use case?