perf-probe: Bugfix and add new options for cache

Hi,

Here is a sereis of patches for enabling "event cache" feature to perf probe. Brendan gives me this cool idea, thanks! :)

In this series, I added following options/features;
- --output option
We can save the probe definition command for given probe-event instead of setting up the local tracing/kprobe_events.

- --no-inlines option
We can avoid searching the inline functions in debuginfo. Usually useful with wildcards since the wildcards will hit a huge amount of probe-points.

- $params special probe argument
$params is expanded to function parameters only, no locally defined variables. This is useful for function-call tracing.

- wildcard support for function name
Wildcard support is the key feature for this idea. Now we can use '*foo*' for function name to define the probe-point.

So by using all of them, we can make an "event cache" file on all functions (except for inlined functions) as below.

  # perf probe --max-probes=100000 --no-inlines -a '* $params' -o event.cache

builds "event.cache" file in which event settings for all function entries, like below;

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

  # wc -l event.cache
  33813 event.cache
  # ls -sh event.cache
  2.3M event.cache
  # ls -sh event.cache.gz
  464K event.cache.gz

For setting up a probe event, you can grep the function name and write it to tracing/kprobe_events, as below;

  # zcat event.cache.gz | \
    grep probe/vfs_symlink > /sys/kernel/debug/tracing/kprobe_events

This can be applied for the remote machine only if the machine runs on completely same kernel binary. Perhaps, we need some helper tool to check it.

Thank you,

---

Masami Hiramatsu (6):
[BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance [DOC] perf-probe: Update perf-probe document
perf-probe: Add --output option to write commands in a standard file
perf-probe: Add --no-inlines option to avoid searching inline functions
perf-probe: Support $params special probe argument
perf-probe: Support glob wildcards for function name

 tools/perf/Documentation/perf-probe.txt |   25 ++++++++++
 tools/perf/builtin-probe.c              |   32 +++++++++++++
 tools/perf/util/dwarf-aux.c             |   31 +++++++++++++
 tools/perf/util/dwarf-aux.h             |    6 +++
 tools/perf/util/probe-event.c           |   73 +++++++++++++++++++++++--------
 tools/perf/util/probe-event.h           |    4 +-
 tools/perf/util/probe-finder.c          |   74 +++++++++++++++++++------------
 tools/perf/util/probe-finder.h          |    6 ++-
 tools/perf/util/util.h                  |    4 ++
 9 files changed, 202 insertions(+), 53 deletions(-)

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Quoted 39 lines

Hi,

Here is a sereis of patches for enabling "event cache" feature to perf probe. Brendan gives me this cool idea, thanks! :)

In this series, I added following options/features;
- --output option
We can save the probe definition command for given probe-event instead of setting up the local tracing/kprobe_events.

- --no-inlines option
We can avoid searching the inline functions in debuginfo. Usually useful with wildcards since the wildcards will hit a huge amount of probe-points.

- $params special probe argument
$params is expanded to function parameters only, no locally defined variables. This is useful for function-call tracing.

- wildcard support for function name
Wildcard support is the key feature for this idea. Now we can use '*foo*' for function name to define the probe-point.

So by using all of them, we can make an "event cache" file on all functions (except for inlined functions) as below.

  # perf probe --max-probes=100000 --no-inlines -a '* $params' -o event.cache

builds "event.cache" file in which event settings for all function entries, like below;

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

Quoted 3 lines

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

How do you validate that the cache can be used against some kernel? I.e. is this that the user has to do? Isn't this prone to errors?

Perhaps you could pick the build-id and store it into the event cache file, in the first lines, somethings like:

[acme@zoo ~]$ printf "buildid: %s\n" $(perf buildid-list --kernel)
buildid: a4cacca49391fc4f42ac8f58990f4e97042efae8

[acme@zoo ~]$ printf "buildid: %s\n" $(perf buildid-list --kernel)
buildid: a4cacca49391fc4f42ac8f58990f4e97042efae8

Maybe this would be nice to have integrated with 'perf archive' somehow and then store this into ~/.debug/[probe]/<BUILDID>/dso-name

where dso-name would be [kernel] for the kernel and the full path for userspace stuff, and then when adding a new probe we would look there for a pre-built/cached event definition, only looking for the debuginfo (which is done using the build-id already, right) and would insert the probe definitions there, etc.

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

Great stuff, keep it up!

- Arnaldo

Quoted 42 lines

  # wc -l event.cache
  33813 event.cache
  # ls -sh event.cache
  2.3M event.cache
  # ls -sh event.cache.gz
  464K event.cache.gz

For setting up a probe event, you can grep the function name and write it to tracing/kprobe_events, as below;

  # zcat event.cache.gz | \
    grep probe/vfs_symlink > /sys/kernel/debug/tracing/kprobe_events

This can be applied for the remote machine only if the machine runs on completely same kernel binary. Perhaps, we need some helper tool to check it.

Thank you,

---

Masami Hiramatsu (6):
[BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance [DOC] perf-probe: Update perf-probe document
perf-probe: Add --output option to write commands in a standard file
perf-probe: Add --no-inlines option to avoid searching inline functions
perf-probe: Support $params special probe argument
perf-probe: Support glob wildcards for function name

 tools/perf/Documentation/perf-probe.txt |   25 ++++++++++
 tools/perf/builtin-probe.c              |   32 +++++++++++++
 tools/perf/util/dwarf-aux.c             |   31 +++++++++++++
 tools/perf/util/dwarf-aux.h             |    6 +++
 tools/perf/util/probe-event.c           |   73 +++++++++++++++++++++++--------
 tools/perf/util/probe-event.h           |    4 +-
 tools/perf/util/probe-finder.c          |   74 +++++++++++++++++++------------
 tools/perf/util/probe-finder.h          |    6 ++-
 tools/perf/util/util.h                  |    4 ++
 9 files changed, 202 insertions(+), 53 deletions(-)

Quoted 5 lines

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 12 lines · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Quoted 11 lines

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

Quoted 2 lines

How do you validate that the cache can be used against some kernel? I.e. is this that the user has to do? Isn't this prone to errors?

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

Quoted 2 lines

Perhaps you could pick the build-id and store it into the event cache file, in the first lines, somethings like:

Agreed, build-id should be the best way to check that.

For kprobes, user can easy to get and compare it with local one as below :)
----
RLOGIN=root@$REMOTE
rid=ssh $RLOGIN "od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '" lid=od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '
if [ $rid != $lid ]; then
echo "Error: Build-id mis-matched!"
exit 1;
fi
echo "Setting up $EVENTNAME at $REMOTE"
zcat event.cache.gz | grep $EVENTNAME |\
ssh $RLOGIN "tee -a /sys/kernel/debug/tracing/kprobe_events" echo "Done"
----

With this script, you don't need to install perf at remote hosts. (This is what enterprise people called "agent-less")

Quoted 8 lines

[acme@zoo ~]$ printf "buildid: %s\n" $(perf buildid-list --kernel)
buildid: a4cacca49391fc4f42ac8f58990f4e97042efae8

[acme@zoo ~]$ printf "buildid: %s\n" $(perf buildid-list --kernel)
buildid: a4cacca49391fc4f42ac8f58990f4e97042efae8

Maybe this would be nice to have integrated with 'perf archive' somehow and then store this into ~/.debug/[probe]/<BUILDID>/dso-name

Quoted 5 lines

where dso-name would be [kernel] for the kernel and the full path for userspace stuff, and then when adding a new probe we would look there for a pre-built/cached event definition, only looking for the debuginfo (which is done using the build-id already, right) and would insert the probe definitions there, etc.

This will be good for SDT too. Perhaps, both of SDT and cached probes should share the same file.

Quoted 5 lines

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

Yeah, querying it from the BUILDID database by using a pair of remote build-id and the binary path is a good feature.

Quoted 4 lines

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

OK, perhaps, that is also good to SDT series at last.

Em Mon, Nov 03, 2014 at 09:11:18PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines

Quoted 12 lines · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Quoted 11 lines

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

Quoted 6 lines

Quoted 2 lines

How do you validate that the cache can be used against some kernel? I.e. is this that the user has to do? Isn't this prone to errors?

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

No, it is not necessarily safe.

What if you specify function foo() that has address 0x1234 for kernel v3.16 and then run the cached probe on kernel v3.18 and on that kernel the function foo() maps to address 0x2345 and function bar() instead maps to address 0x1234? Oops.

The build-id was designed to uniquely identify a DSO, we need to use it.

I think that at some point not using it should be left to a, in systemtap parlance, "guru" mode, with tooling warning profusely when build ids are not available and requiring even more forcing when it doesn't matches.

Quoted 20 lines

Quoted 2 lines

Perhaps you could pick the build-id and store it into the event cache file, in the first lines, somethings like:

Agreed, build-id should be the best way to check that.

For kprobes, user can easy to get and compare it with local one as below :)
----
RLOGIN=root@$REMOTE
rid=ssh $RLOGIN "od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '" lid=od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '
if [ $rid != $lid ]; then
echo "Error: Build-id mis-matched!"
exit 1;
fi
echo "Setting up $EVENTNAME at $REMOTE"
zcat event.cache.gz | grep $EVENTNAME |\
ssh $RLOGIN "tee -a /sys/kernel/debug/tracing/kprobe_events" echo "Done"
----

Quoted 2 lines

With this script, you don't need to install perf at remote hosts. (This is what enterprise people called "agent-less")

This is only sufficient (and a cool feature!) if you will immediataly use the cached info (i.e. using just two systems: one development machine, with all debugging info, devel tools, etc, and the other the machine to observe, that is bare bones, no debugging info, etc)), but the moment you store that event.cache.gz (that has no build id embedded from what I can see from the above example) then you lose the build id for those cached events.

You need to tightly associate whatever symbol resolution is done with the build id, at symbol resolution/caching time.

Then, before using the cached symbol resolution, we need to check if the target kernel/DSO build id is the same as the cached symbol kernel/DSO build id.

Right, what is in ~/.debug/ may be used by multiple tools, just like debuginfo files are, by keying the content by its build id.

And also by having separate subdirectory trees for different kinds of symbol information, i.e. the ~/.debug/.build-id/ links may point to either ELF files or to kallsyms data. What we are discussing here is to make it also point to the [ku]probes_tracer cached probes files.

The way that DSO files are cached may by the same that you end up adding the [ku]probes_tracer cached files, take a look at the example of caching for the '/usr/bin/gcc' DSO on a test maachine here at my home lab:

This solves the problem with debuginfo packages where we can't have multiple debuginfo packages installed, as well as for files that didn't came from debuginfo files.

[root@zoo ~]# perf buildid-cache --hell
Error: unknown option `hell'

usage: perf buildid-cache [<options>]

    -a, --add <file list>
                          file(s) to add
    -k, --kcore <file>    kcore file to add
    -r, --remove <file list>
                          file(s) to remove
    -M, --missing <file>  to find missing build ids in the cache
    -f, --force           don't complain, do it
    -u, --update <file list>
                          file(s) to update
    -v, --verbose         be more verbose

[root@zoo ~]#

Already has support for yet another of content: kcore files, its just a matter of adding one more:

   perf buildid-cache --probe 

:-)

Quoted 9 lines

Quoted 6 lines

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

Yeah, querying it from the BUILDID database by using a pair of remote build-id and the binary path is a good feature.

Quoted 6 lines

Quoted 4 lines

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

OK, perhaps, that is also good to SDT series at last.

Sure thing!

Quoted 8 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 03, 2014 at 09:11:18PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Quoted 1 line · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

No, it is not necessarily safe.

Quoted 4 lines

What if you specify function foo() that has address 0x1234 for kernel v3.16 and then run the cached probe on kernel v3.18 and on that kernel the function foo() maps to address 0x2345 and function bar() instead maps to address 0x1234? Oops.

In that case user just trace bar() instead of foo(). Of course it's not correct, but shouldn't break the kernel (if the kernel is broken, it is a bug of kprobes).

Quoted 6 lines

The build-id was designed to uniquely identify a DSO, we need to use it.

I think that at some point not using it should be left to a, in systemtap parlance, "guru" mode, with tooling warning profusely when build ids are not available and requiring even more forcing when it doesn't matches.

But it is not necessarily everyone uses perf probe to set up the probe events(because it is a part of ftrace), as we can see in the Brendan's scripts.

I think, at least what we need is clarifying how can they ensure build-id before setting the probe events. I'd like to give them options with knowledge instead of forcing by tools.

Quoted 23 lines

Quoted 20 lines

Quoted 2 lines

Perhaps you could pick the build-id and store it into the event cache file, in the first lines, somethings like:

Agreed, build-id should be the best way to check that.

For kprobes, user can easy to get and compare it with local one as below :)
----
RLOGIN=root@$REMOTE
rid=ssh $RLOGIN "od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '" lid=od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' '
if [ $rid != $lid ]; then
echo "Error: Build-id mis-matched!"
exit 1;
fi
echo "Setting up $EVENTNAME at $REMOTE"
zcat event.cache.gz | grep $EVENTNAME |\
ssh $RLOGIN "tee -a /sys/kernel/debug/tracing/kprobe_events" echo "Done"
----

Quoted 2 lines

With this script, you don't need to install perf at remote hosts. (This is what enterprise people called "agent-less")

Quoted 6 lines

This is only sufficient (and a cool feature!) if you will immediataly use the cached info (i.e. using just two systems: one development machine, with all debugging info, devel tools, etc, and the other the machine to observe, that is bare bones, no debugging info, etc)), but the moment you store that event.cache.gz (that has no build id embedded from what I can see from the above example) then you lose the build id for those cached events.

Right, in this case, it should be stored with build-id, like below :)

lid=od -j16 -w48 -An -t x1 /sys/kernel/notes | tr -d ' ' perf probe -o - --max-probes=10000 --no-inlines -a '* $params' | \
gzip -c - > $lid.gz

Anyway, this is just a workaround by operation.

Quoted 2 lines

You need to tightly associate whatever symbol resolution is done with the build id, at symbol resolution/caching time.

Agreed,

Quoted 2 lines

Then, before using the cached symbol resolution, we need to check if the target kernel/DSO build id is the same as the cached symbol kernel/DSO build id.

Yeah, but again, some users may not like to install perf to the target system (like embedded system etc.) and also, I, personally, like to avoid introducing server-client networking feature for perftools, since it may open another pandora-box of security... I'd like to use it combining with other tools, like ssh or something like that.

Quoted 8 lines

Right, what is in ~/.debug/ may be used by multiple tools, just like debuginfo files are, by keying the content by its build id.

And also by having separate subdirectory trees for different kinds of symbol information, i.e. the ~/.debug/.build-id/ links may point to either ELF files or to kallsyms data. What we are discussing here is to make it also point to the [ku]probes_tracer cached probes files.

Quoted 4 lines

The way that DSO files are cached may by the same that you end up adding the [ku]probes_tracer cached files, take a look at the example of caching for the '/usr/bin/gcc' DSO on a test maachine here at my home lab:

Thanks, this is useful for me :)

[snip]

Quoted 21 lines

This solves the problem with debuginfo packages where we can't have multiple debuginfo packages installed, as well as for files that didn't came from debuginfo files.

[root@zoo ~]# perf buildid-cache --hell
Error: unknown option `hell'

usage: perf buildid-cache [<options>]

    -a, --add <file list>
                          file(s) to add
    -k, --kcore <file>    kcore file to add
    -r, --remove <file list>
                          file(s) to remove
    -M, --missing <file>  to find missing build ids in the cache
    -f, --force           don't complain, do it
    -u, --update <file list>
                          file(s) to update
    -v, --verbose         be more verbose

Quoted 8 lines

[root@zoo ~]#

Already has support for yet another of content: kcore files, its just a matter of adding one more:

   perf buildid-cache --probe 

:-)

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

  perf probe --query ${remote_buildid}:do_fork
  p:probe/do_fork _text+298722 clone_flags=%di:u64 stack_start=%si:u64 stack_size=%dx:u64 parent_tidptr=%cx:u64 child_tidptr=%r8:u64

  # or perf can set it up directly to local
  perf probe --query-add do_fork

  # or perf can set it up directly to local
  perf probe --query-add ${remote_buildid}:do_fork

Added new event:
probe:do_fork (on do_fork with clone_flags satck_start stack_size parent_tidptr child_tidptr)

You can now use it in all perf tools, such as:

perf record -e probe:do_fork -aR sleep 1

What would you think about this? :)

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

Yeah, querying it from the BUILDID database by using a pair of remote build-id and the binary path is a good feature.

Quoted 4 lines

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

Quoted 4 lines

Quoted 1 line

OK, perhaps, that is also good to SDT series at last.

Sure thing!

Thank you!

Em Tue, Nov 04, 2014 at 01:36:31PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines

Quoted 13 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 03, 2014 at 09:11:18PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Quoted 1 line · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

No, it is not necessarily safe.

What if you specify function foo() that has address 0x1234 for kernel v3.16 and then run the cached probe on kernel v3.18 and on that kernel the function foo() maps to address 0x2345 and function bar() instead maps to address 0x1234? Oops.

Quoted 3 lines

In that case user just trace bar() instead of foo(). Of course it's not correct, but shouldn't break the kernel (if the kernel is broken, it is a bug of kprobes).

I.e. no crashes, just misleading information :-\

Quoted 10 lines

Quoted 6 lines

The build-id was designed to uniquely identify a DSO, we need to use it.

I think that at some point not using it should be left to a, in systemtap parlance, "guru" mode, with tooling warning profusely when build ids are not available and requiring even more forcing when it doesn't matches.

But it is not necessarily everyone uses perf probe to set up the probe events(because it is a part of ftrace), as we can see in the Brendan's scripts.

Right, If I implied that some particular tool should be used, sorry about that, what I wanted to get accross was that the information that allows users or tools to make sure there is no mismatch between the cached probes and the target kernel is collected at cached probe creating time and available at target use time.

Quoted 4 lines

I think, at least what we need is clarifying how can they ensure build-id before setting the probe events. I'd like to give them options with knowledge instead of forcing by tools.

Right, so we need to have the build-id as part of the cache format, perhaps as the first line, starting with a comment (#), that way the user can use whatever way he has at its disposal to check that the running kernel build-id is the same as the one on that comment. Using that script you provided, that uses just things that are on the machine (od, /sys/kernel/notes).

Quoted 8 lines

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

To follow the existing standard this would instead go to:

Quoted 2 lines

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

Quoted 6 lines

  perf probe --query ${remote_buildid}:do_fork
  p:probe/do_fork _text+298722 clone_flags=%di:u64 stack_start=%si:u64 stack_size=%dx:u64 parent_tidptr=%cx:u64 child_tidptr=%r8:u64

  # or perf can set it up directly to local
  perf probe --query-add do_fork

You missed the build id above, no? I.e. it would be:

Quoted 2 lines

  # or perf can set it up directly to local
  perf probe --query-add ${remote_buildid}:do_fork

Right? and that would be the equivalent to:

perf probe --query ${remote_buildid}:do_fork > /sys/kernel/debug/tracing/kprobe_events

No?

And do you intend to generate that script above that checks the build id, tees into /sys/kernel/debug/tracing/kprobe_events, etc as part of this process?

I.e. the process would be, as you describe above plus telling the user that he/she then should unconpress the script and run it.

Scripts like Brendan's would do whatever they want with it of course, but the default end result of 'perf probe' would be usable straight away and doing the build id checking without requiring any extra tooling to be available at the target machine.

Quoted 9 lines

Added new event:
probe:do_fork (on do_fork with clone_flags satck_start stack_size parent_tidptr child_tidptr)

You can now use it in all perf tools, such as:

perf record -e probe:do_fork -aR sleep 1

What would you think about this? :)

Cool feature! :-)

Quoted 14 lines

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

Yeah, querying it from the BUILDID database by using a pair of remote build-id and the binary path is a good feature.

Quoted 4 lines

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

OK, perhaps, that is also good to SDT series at last.

Sure thing!

Thank you!

You're welcome!

- Arnaldo

Quoted 10 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 04, 2014 at 01:36:31PM +0900, Masami Hiramatsu escreveu:

Quoted 8 lines

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

Quoted 4 lines

To follow the existing standard this would instead go to:

Quoted 2 lines

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

Ah, I see. so you meant adding a top-level .debug/probes/ dir. But in that case, shouldn't we change .debug/.buildid/bu/ildid to .debug/probes/.buildid/bu/ildid ?

Quoted 10 lines

Quoted 7 lines

  perf probe --query ${remote_buildid}:do_fork
  p:probe/do_fork _text+298722 clone_flags=%di:u64 stack_start=%si:u64 stack_size=%dx:u64 parent_tidptr=%cx:u64 child_tidptr=%r8:u64

  # or perf can set it up directly to local
  perf probe --query-add do_fork

You missed the build id above, no? I.e. it would be:

Quoted 2 lines

Quoted 2 lines

  # or perf can set it up directly to local
  perf probe --query-add ${remote_buildid}:do_fork

No, since this command set the event to local machine, perf-probe should check the local build-id and query the appropriate event from the cache.
# BTW, maybe we'd better use perf probe --add '$do_fork' (calls # "cache of do_fork") instead of long --query-add. :)

Quoted 16 lines · Masami Hiramatsu

Quoted 14 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 04, 2014 at 01:36:31PM +0900, Masami Hiramatsu escreveu:

Quoted 8 lines

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

To follow the existing standard this would instead go to:

Quoted 2 lines

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

Quoted 3 lines

Ah, I see. so you meant adding a top-level .debug/probes/ dir. But in that case, shouldn't we change .debug/.buildid/bu/ildid to .debug/probes/.buildid/bu/ildid ?

Either is fine to me. But my concern is that it might be bloated as system/package update is going on, so we need to control it somehow. That's why I suggested the probe-cache command.

Quoted 8 lines

Quoted 8 lines

Quoted 5 lines

  perf probe --query ${remote_buildid}:do_fork
  p:probe/do_fork _text+298722 clone_flags=%di:u64 stack_start=%si:u64 stack_size=%dx:u64 parent_tidptr=%cx:u64 child_tidptr=%r8:u64

  # or perf can set it up directly to local
  perf probe --query-add do_fork

You missed the build id above, no? I.e. it would be:

Quoted 8 lines

Quoted 2 lines

Quoted 2 lines

  # or perf can set it up directly to local
  perf probe --query-add ${remote_buildid}:do_fork

No, since this command set the event to local machine, perf-probe should check the local build-id and query the appropriate event from the cache.
# BTW, maybe we'd better use perf probe --add '$do_fork' (calls # "cache of do_fork") instead of long --query-add. :)

It should take care of uprobe case too. So simple do_fork should have group/event or provider/marker form instead so that it can help to find which binary defines the cached event. Maybe we also need to keep a event-to-binary table and then check (current?) build-id somehow to identify correct event to be used.

Also this function entry level event cache can be used with uprobes..

Quoted 15 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 04, 2014 at 01:36:31PM +0900, Masami Hiramatsu escreveu:

Quoted 14 lines

Quoted 13 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 03, 2014 at 09:11:18PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Quoted 1 line · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

No, it is not necessarily safe.

What if you specify function foo() that has address 0x1234 for kernel v3.16 and then run the cached probe on kernel v3.18 and on that kernel the function foo() maps to address 0x2345 and function bar() instead maps to address 0x1234? Oops.

Quoted 5 lines

Quoted 3 lines

In that case user just trace bar() instead of foo(). Of course it's not correct, but shouldn't break the kernel (if the kernel is broken, it is a bug of kprobes).

I.e. no crashes, just misleading information :-\

Right.

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

The build-id was designed to uniquely identify a DSO, we need to use it.

I think that at some point not using it should be left to a, in systemtap parlance, "guru" mode, with tooling warning profusely when build ids are not available and requiring even more forcing when it doesn't matches.

But it is not necessarily everyone uses perf probe to set up the probe events(because it is a part of ftrace), as we can see in the Brendan's scripts.

Quoted 5 lines

Right, If I implied that some particular tool should be used, sorry about that, what I wanted to get accross was that the information that allows users or tools to make sure there is no mismatch between the cached probes and the target kernel is collected at cached probe creating time and available at target use time.

Yes, and if user setting probes via perf, the perf must ensure that it picks up the correct cache by using build-id. If someone wants to use other tools, he/she must ensure it. We just give a information how to check that :)

Quoted 10 lines

Quoted 3 lines

I think, at least what we need is clarifying how can they ensure build-id before setting the probe events. I'd like to give them options with knowledge instead of forcing by tools.

Right, so we need to have the build-id as part of the cache format, perhaps as the first line, starting with a comment (#), that way the user can use whatever way he has at its disposal to check that the running kernel build-id is the same as the one on that comment. Using that script you provided, that uses just things that are on the machine (od, /sys/kernel/notes).

Ah, that's a good idea :)
So, with such build-id comment line, would you think we can have an --output option? Or we'd better moving onto the .debug/ cache file?

What I'm thinking about this feature is to make a compact and reduced function-entry level probe cache while building the kernel (as a part of kbuild), so that we can deploy the stripped kernel and the cache to remote machines.

[snip]

Quoted 9 lines

Quoted 8 lines

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

Quoted 4 lines

To follow the existing standard this would instead go to:

Quoted 2 lines

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

Ah, I see. so you meant adding a top-level .debug/probes/ dir. But in that case, shouldn't we change .debug/.buildid/bu/ildid to .debug/probes/.buildid/bu/ildid ?

Quoted 8 lines

Quoted 5 lines

  perf probe --query ${remote_buildid}:do_fork
  p:probe/do_fork _text+298722 clone_flags=%di:u64 stack_start=%si:u64 stack_size=%dx:u64 parent_tidptr=%cx:u64 child_tidptr=%r8:u64

  # or perf can set it up directly to local
  perf probe --query-add do_fork

You missed the build id above, no? I.e. it would be:

Quoted 2 lines

Quoted 2 lines

  # or perf can set it up directly to local
  perf probe --query-add ${remote_buildid}:do_fork

No, since this command set the event to local machine, perf-probe should check the local build-id and query the appropriate event from the cache.
# BTW, maybe we'd better use perf probe --add '$do_fork' (calls # "cache of do_fork") instead of long --query-add. :)

Quoted 10 lines

Right? and that would be the equivalent to:

perf probe --query ${remote_buildid}:do_fork > /sys/kernel/debug/tracing/kprobe_events

No?

And do you intend to generate that script above that checks the build id, tees into /sys/kernel/debug/tracing/kprobe_events, etc as part of this process?

No, not yet. But that sounds nice :) Thanks!

Quoted 8 lines

I.e. the process would be, as you describe above plus telling the user that he/she then should unconpress the script and run it.

Scripts like Brendan's would do whatever they want with it of course, but the default end result of 'perf probe' would be usable straight away and doing the build id checking without requiring any extra tooling to be available at the target machine.

Agreed, with .debug/ archive, perf-probe should pick and set event correctly without any other tools.

Thank you!

Quoted 37 lines

Quoted 9 lines

Added new event:
probe:do_fork (on do_fork with clone_flags satck_start stack_size parent_tidptr child_tidptr)

You can now use it in all perf tools, such as:

perf record -e probe:do_fork -aR sleep 1

What would you think about this? :)

Cool feature! :-)

Quoted 14 lines

Quoted 14 lines

Quoted 13 lines

Quoted 5 lines

Then, later, one would use 'perf archive' passing some keys (or a perf.data file, like done nowadays to pick the files in ~/.debug for dsos that had hits on the specified perf.data file) to get the cached values to use on some other machine, to avoid having to use the debuginfo files there.

Yeah, querying it from the BUILDID database by using a pair of remote build-id and the binary path is a good feature.

Quoted 4 lines

I.e. in summary I think that the format is ok, but we need to have this inside the ~/.debug hierarchy so that we can make sure that we use the right probe definition, one that matches the DSOs being used (the kernel or some other userspace binary).

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

OK, perhaps, that is also good to SDT series at last.

Sure thing!

Thank you!

You're welcome!

- Arnaldo

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Em Wed, Nov 05, 2014 at 01:22:46AM +0900, Masami Hiramatsu escreveu:

Quoted 20 lines

Quoted 19 lines · Arnaldo Carvalho de Melo

Em Tue, Nov 04, 2014 at 01:36:31PM +0900, Masami Hiramatsu escreveu:

Quoted 18 lines

Quoted 7 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 03, 2014 at 09:11:18PM +0900, Masami Hiramatsu escreveu:

Quoted 5 lines

Quoted 1 line · Arnaldo Carvalho de Melo

Em Fri, Oct 31, 2014 at 02:51:29PM -0400, Masami Hiramatsu escreveu:

Actually, kprobe event itself can reject command if the given address is not in the kernel text nor instruction boundary (perhaps, uprobes may have a problem...), so for the kernel level, it is safe.

Quoted 6 lines

No, it is not necessarily safe.

What if you specify function foo() that has address 0x1234 for kernel v3.16 and then run the cached probe on kernel v3.18 and on that kernel the function foo() maps to address 0x2345 and function bar() instead maps to address 0x1234? Oops.

In that case user just trace bar() instead of foo(). Of course it's not correct, but shouldn't break the kernel (if the kernel is broken, it is a bug of kprobes).

Quoted 3 lines

Quoted 1 line

I.e. no crashes, just misleading information :-\

Right.

The kernel doesn't crashes, just the user, after scratching his head trying to make sense of wrong information :-P

Digressing: is there some kprobe_tracer_fuzzer out there?

Quoted 33 lines

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

The build-id was designed to uniquely identify a DSO, we need to use it.

I think that at some point not using it should be left to a, in systemtap parlance, "guru" mode, with tooling warning profusely when build ids are not available and requiring even more forcing when it doesn't matches.

But it is not necessarily everyone uses perf probe to set up the probe events(because it is a part of ftrace), as we can see in the Brendan's scripts.

Quoted 5 lines

Right, If I implied that some particular tool should be used, sorry about that, what I wanted to get accross was that the information that allows users or tools to make sure there is no mismatch between the cached probes and the target kernel is collected at cached probe creating time and available at target use time.

Yes, and if user setting probes via perf, the perf must ensure that it picks up the correct cache by using build-id. If someone wants to use other tools, he/she must ensure it. We just give a information how to check that :)

Quoted 10 lines

Quoted 3 lines

I think, at least what we need is clarifying how can they ensure build-id before setting the probe events. I'd like to give them options with knowledge instead of forcing by tools.

Right, so we need to have the build-id as part of the cache format, perhaps as the first line, starting with a comment (#), that way the user can use whatever way he has at its disposal to check that the running kernel build-id is the same as the one on that comment. Using that script you provided, that uses just things that are on the machine (od, /sys/kernel/notes).

Quoted 3 lines

Ah, that's a good idea :)
So, with such build-id comment line, would you think we can have an --output option? Or we'd better moving onto the .debug/ cache file?

Well, these are two separate things: The ./debug cache file is for repeated use of the same probe on the same machine, say, across reboots or in tools that script using perf probe to add some probe, then remove the probes at exit, using it multiple times would provide a after caching, as no ELF DWARF parsing would be involved after we cached it.

As well for when using 'perf archive' to reuse the probe definitions using 'perf probe' on the target machine, where no -debuginfo packages (or binaries with DWARF) would be available.

I.e. both having the comment on the cache file with the build id and having it stored in ~/.debug/ have values for different workloads or different ways of doing the same thing.

Quoted 4 lines

What I'm thinking about this feature is to make a compact and reduced function-entry level probe cache while building the kernel (as a part of kbuild), so that we can deploy the stripped kernel and the cache to remote machines.

Which is a cool feature as well! :-)

Quoted 11 lines

[snip]

Quoted 9 lines

Quoted 8 lines

OK, I agree using .debug/.buildid/ to store caches. Here is what I'm thinking,

  # this makes caches for given pattern instead of adding probes.
  perf probe --cache '* $params'

  # the cache is stored in .debug/.buildid/<buildid>.probe
  # the cache entry can be queried by buildid and eventname

Quoted 8 lines

Quoted 4 lines

To follow the existing standard this would instead go to:

Quoted 2 lines

  # the cache is stored in .debug/probes/path/to/dso/name/buildid
  # And can be found via its buildid link .debug/.buildid/bu/ildid -> ../../probes/path/to/dso/name/buildid

Ah, I see. so you meant adding a top-level .debug/probes/ dir. But in that case, shouldn't we change .debug/.buildid/bu/ildid to .debug/probes/.buildid/bu/ildid ?

Humm, understood, perhaps we should have a subdir for probes, i.e.:

# And can be found via its buildid link .debug/.buildid/probes/bu/ildid -> ../../probes/path/to/dso/name/probes/buildid

So as not to clash in both cases with the ELF file.

Which leads to a problem when we have both vmlinux and kallsyms, which, right now, is not possible, i.e. if we add both a vmlinux and a kallsyms file for a kernel, the one that comes last will have the

.debug/.buildid/bu/ildid

link, have to fix that so that we first look for the ELF file in the cache, and if it fails, fallback to the kallsyms file, if available, i.e. give preference to the richer option, and in some cases, like for annotation, the only one that matters.

[root@zoo ~]# perf buildid-cache --hell
Error: unknown option `hell'

usage: perf buildid-cache [<options>]

    -a, --add <file list>
                          file(s) to add
    -k, --kcore <file>    kcore file to add
    -r, --remove <file list>
                          file(s) to remove
    -M, --missing <file>  to find missing build ids in the cache
    -f, --force           don't complain, do it
    -u, --update <file list>
                          file(s) to update
    -v, --verbose         be more verbose

[root@zoo ~]#

Already has support for yet another of content: kcore files, its just a matter of adding one more:

   perf buildid-cache --probe 

Hi,

Here is a sereis of patches for enabling "event cache" feature to perf probe. Brendan gives me this cool idea, thanks! :)

In this series, I added following options/features;
- --output option
We can save the probe definition command for given probe-event instead of setting up the local tracing/kprobe_events.

- --no-inlines option
We can avoid searching the inline functions in debuginfo. Usually useful with wildcards since the wildcards will hit a huge amount of probe-points.

- $params special probe argument
$params is expanded to function parameters only, no locally defined variables. This is useful for function-call tracing.

- wildcard support for function name
Wildcard support is the key feature for this idea. Now we can use '*foo*' for function name to define the probe-point.

So by using all of them, we can make an "event cache" file on all functions (except for inlined functions) as below.

  # perf probe --max-probes=100000 --no-inlines -a '* $params' -o event.cache

builds "event.cache" file in which event settings for all function entries, like below;

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

  # wc -l event.cache
  33813 event.cache
  # ls -sh event.cache
  2.3M event.cache
  # ls -sh event.cache.gz
  464K event.cache.gz

For setting up a probe event, you can grep the function name and write it to tracing/kprobe_events, as below;

  # zcat event.cache.gz | \
    grep probe/vfs_symlink > /sys/kernel/debug/tracing/kprobe_events

This can be applied for the remote machine only if the machine runs on completely same kernel binary. Perhaps, we need some helper tool to check it.

Thank you,

---

Masami Hiramatsu (6):
[BUGFIX] perf-probe: Fix to handle optimized not-inlined but has no instance [DOC] perf-probe: Update perf-probe document
perf-probe: Add --output option to write commands in a standard file
perf-probe: Add --no-inlines option to avoid searching inline functions
perf-probe: Support $params special probe argument
perf-probe: Support glob wildcards for function name

 tools/perf/Documentation/perf-probe.txt |   25 ++++++++++
 tools/perf/builtin-probe.c              |   32 +++++++++++++
 tools/perf/util/dwarf-aux.c             |   31 +++++++++++++
 tools/perf/util/dwarf-aux.h             |    6 +++
 tools/perf/util/probe-event.c           |   73 +++++++++++++++++++++++--------
 tools/perf/util/probe-event.h           |    4 +-
 tools/perf/util/probe-finder.c          |   74 +++++++++++++++++++------------
 tools/perf/util/probe-finder.h          |    6 ++-
 tools/perf/util/util.h                  |    4 ++
 9 files changed, 202 insertions(+), 53 deletions(-)

--
Masami HIRAMATSU
Software Platform Research Dpt. Linux Technology Center Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Hi Masami,

Quoted 31 lines · Masami Hiramatsu

Hi,

Here is a sereis of patches for enabling "event cache" feature to perf probe. Brendan gives me this cool idea, thanks! :)

In this series, I added following options/features;
- --output option
We can save the probe definition command for given probe-event instead of setting up the local tracing/kprobe_events.

- --no-inlines option
We can avoid searching the inline functions in debuginfo. Usually useful with wildcards since the wildcards will hit a huge amount of probe-points.

- $params special probe argument
$params is expanded to function parameters only, no locally defined variables. This is useful for function-call tracing.

- wildcard support for function name
Wildcard support is the key feature for this idea. Now we can use '*foo*' for function name to define the probe-point.

So by using all of them, we can make an "event cache" file on all functions (except for inlined functions) as below.

  # perf probe --max-probes=100000 --no-inlines -a '* $params' -o event.cache

builds "event.cache" file in which event settings for all function entries, like below;

Quoted 7 lines

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

Does this event cache support kernel modules too? AFAIK it can have a different address whenever loaded even on a same kernel.

Quoted 17 lines

This event.cache file will be big (but much smaller than native debuginfo :) ) if your kernel have many option embedded. Anyway, you can compress it too.

  # wc -l event.cache
  33813 event.cache
  # ls -sh event.cache
  2.3M event.cache
  # ls -sh event.cache.gz
  464K event.cache.gz

For setting up a probe event, you can grep the function name and write it to tracing/kprobe_events, as below;

  # zcat event.cache.gz | \
    grep probe/vfs_symlink > /sys/kernel/debug/tracing/kprobe_events

Quoted 3 lines

This can be applied for the remote machine only if the machine runs on completely same kernel binary. Perhaps, we need some helper tool to check it.

While it's useful for "agent-less" systems, I think we also need to have a simple way to apply it with perf tools.

Quoted 19 lines · Namhyung Kim

Hi Masami,

Quoted 8 lines · Masami Hiramatsu

So by using all of them, we can make an "event cache" file on all functions (except for inlined functions) as below.

  # perf probe --max-probes=100000 --no-inlines -a '* $params' -o event.cache

builds "event.cache" file in which event settings for all function entries, like below;

Quoted 7 lines

p:probe/reset_early_page_tables _text+12980741 p:probe/copy_bootdata _text+12980830 real_mode_data=%di:u64 p:probe/exit_amd_microcode _text+14692680 p:probe/early_make_pgtable _text+12981274 address=%di:u64 p:probe/x86_64_start_reservations _text+12981700 real_mode_data=%di:u64 p:probe/x86_64_start_kernel _text+12981744 real_mode_data=%di:u64 p:probe/reserve_ebda_region _text+12982117

Quoted 2 lines

Does this event cache support kernel modules too? AFAIK it can have a different address whenever loaded even on a same kernel.

Yes, for the modules perf probe uses target function symbol directly instead of _text.

----
perf probe -m xfs -o - -q -a xfs_acl_exists
p:probe/xfs_acl_exists xfs:xfs_acl_exists+0
----