perf/sdt: SDT events listing/probing

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

            event_hash       sdt_note
            |---------|   ----------------
            |         |   |   file_ptr   |==> container file_sdt_ent
key = 129 =>| hlist ==|===|=> event_list=|==> to sdt notes hashed to
       	    |         |   |   name       |    same entry
            |---------|   |   provider   |
            |	      |   |   note_list==|==> to other notes in the
key = 130 =>| hlist   |   ---------------     same file
            |---------|

The entry at that key in event_hash contains a list of SDT notes hashed to the same entry. It compares the name and provider to see if that is the SDT note we are looking for. If yes, find out the file that contains this SDT note. There is a file_ptr pointer embedded in this note which points to the struct file_sdt_ent contained in the file_hash. From "file_sdt_ent" we will find out the file name.
Convert this sdt note into a perf event and then write this into uprobe_events file to be able to record the event.
Then, corresponding entries are added to uprobe_events file for the SDT events.
After recording is done, these events are silently deleted from uprobe_events file. The uprobe_events file is present in debugfs/tracing directory.

To support the addition and deletion of SDT events to/from uprobe_events file, a record_sdt struct is maintained which has the event data.

An example usage:

# perf record -e %libc:setjmp -aR sleep 10
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.277 MB perf.data (~12103 samples) ]

# perf report --stdio
# To display the perf.data header info, please use --header/--header-only
#
# Samples: 1  of event 'libc:setjmp'
# Event count (approx.): 1
#
# Overhead  Command  Shared Object  Symbol
# ........  .......  .............  ...............
#
   100.00%  sleep    libc-2.16.so   [.] __sigsetjmp

Signed-off-by Hemant Kumar <[email protected]>

tools/perf/builtin-record.c +7 -0

+/* Session specific to SDT tracing */
+struct record_sdt {
+	bool    sdt;    /* is this SDT event tracing? */
+	char    *str;   /* hold the event name */
+} rec_sdt;
+
+int trace_sdt_event(const char *str)

tools/perf/util/probe-event.c +3 -0

+	pp->offset = sev->note->addr.a64[0];
+	return 0;
+}

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 5 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

Quoted 7 lines

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Quoted 45 lines

             event_hash       sdt_note
             |---------|   ----------------
             |         |   |   file_ptr   |==> container file_sdt_ent
key = 129 =>| hlist ==|===|=> event_list=|==> to sdt notes hashed to
        	    |         |   |   name       |    same entry
             |---------|   |   provider   |
             |	      |   |   note_list==|==> to other notes in the
key = 130 =>| hlist   |   ---------------     same file
             |---------|

The entry at that key in event_hash contains a list of SDT notes hashed to the same entry. It compares the name and provider to see if that is the SDT note we are looking for. If yes, find out the file that contains this SDT note. There is a file_ptr pointer embedded in this note which points to the struct file_sdt_ent contained in the file_hash. From "file_sdt_ent" we will find out the file name.
Convert this sdt note into a perf event and then write this into uprobe_events file to be able to record the event.
Then, corresponding entries are added to uprobe_events file for the SDT events.
After recording is done, these events are silently deleted from uprobe_events file. The uprobe_events file is present in debugfs/tracing directory.

To support the addition and deletion of SDT events to/from uprobe_events file, a record_sdt struct is maintained which has the event data.

An example usage:

# perf record -e %libc:setjmp -aR sleep 10
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.277 MB perf.data (~12103 samples) ]

# perf report --stdio
# To display the perf.data header info, please use --header/--header-only
#
# Samples: 1  of event 'libc:setjmp'
# Event count (approx.): 1
#
# Overhead  Command  Shared Object  Symbol
# ........  .......  .............  ...............
#
    100.00%  sleep    libc-2.16.so   [.] __sigsetjmp

Signed-off-by Hemant Kumar <[email protected]>

[SNIP]

Quoted 7 lines

tools/perf/builtin-record.c +7 -0

+/* Session specific to SDT tracing */
+struct record_sdt {
+	bool    sdt;    /* is this SDT event tracing? */
+	char    *str;   /* hold the event name */
+} rec_sdt;
+
+int trace_sdt_event(const char *str)

tools/perf/builtin-record.c struct record { · line 66

Quoted 10 lines

tools/perf/builtin-record.c +10 -0

+{
+	int ret = 0;
+
+	rec_sdt.sdt = true;
+	rec_sdt.str = strdup(str);
+	if (!rec_sdt.str)
+		return -ENOMEM;
+	ret = event_hash_list__lookup(str);
+	return ret;
+}

Hmm.. does it support multiple SDT events recorded in a session like:

   # perf record -e %libc:setjmp -e %libc:longjmp -aR sleep 10

?

[SNIP]

tools/perf/util/probe-event.c static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev) · line 1135

Quoted 13 lines

tools/perf/util/probe-event.c +13 -0

+/* Parse an SDT event */
+static int parse_perf_sdt_event(struct perf_sdt_event *sev,
+				struct perf_probe_event *pev)
+{
+	struct perf_probe_point *pp = &pev->point;
+
+	pev->uprobes = true;
+	pev->sdt = true;
+	pev->event = strdup(sev->note->name);
+	if (pev->event == NULL)
+		return -ENOMEM;
+	pev->group = strdup(sev->note->provider);
+	if (pev->event == NULL)

s/event/group/

tools/perf/util/probe-event.c static int parse_perf_probe_point(char *arg, struct perf_probe_event *pev) · line 1142

Quoted 7 lines

tools/perf/util/probe-event.c +7 -0

+		return -ENOMEM;
+
+	pp->file = strdup(sev->file_name);
+	if (pp->file == NULL)
+		return -ENOMEM;
+
+	pp->function = strdup(sev->note->name);

Missing NULL check. Also you need to free prior allocations in case of error.

Thanks,
Namhyung

Quoted 3 lines

tools/perf/util/probe-event.c +3 -0

+	pp->offset = sev->note->addr.a64[0];
+	return 0;
+}

Hi Namhyung,

Quoted 24 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 5 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

Quoted 7 lines

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

Hi,

Quoted 27 lines · Hemant Kumar

Hi Namhyung,

Quoted 24 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 5 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

Quoted 7 lines

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps. I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is

to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands? This question should be involving your series too.

Quoted 11 lines

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Quoted 5 lines

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

How about failing if the provider name is not unique unless user gives the actual binary path?

Hi Masami,

Quoted 35 lines

Hi,

Quoted 31 lines · Hemant Kumar

Hi Namhyung,

Quoted 28 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 6 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

Quoted 10 lines

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events
from the cache, its not of much use when it comes to probing the actual
SDT event, because the key to this hash list is a file name and not the
SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

Quoted 5 lines

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps. I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is

Like a single cache to manage all the events for that file? How do we distinguish between the events as we will be having perf record to read SDT events from this cache?

Quoted 3 lines

to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

This approach looks convenient.

Quoted 6 lines

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands?
This question should be involving your series too.

I think, we need not have multiple sub-commands to manage the cache. We can consolidate the cache management of probe events (including SDT events) to a single command.

Quoted 18 lines

Quoted 17 lines

Quoted 11 lines

Quoted 7 lines

To avoid this problem, we can create another hash list "event_hash" list
which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list. The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

Quoted 4 lines

How about failing if the provider name is not unique unless user gives the actual binary path?

You mean something like this:

# perf record -e %provider @/path/to/file ...?

Quoted 5 lines · Hemant Kumar

Quoted 5 lines

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands?
This question should be involving your series too.

Quoted 3 lines

I think, we need not have multiple sub-commands to manage the cache. We can consolidate the cache management of probe events (including SDT events) to a single command.

Agreed. maybe perf-cache --buildid/--sdt/--probe would be good.

Hi,

Quoted 24 lines · Hemant Kumar

Hi Namhyung,

Quoted 21 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 12 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

Quoted 3 lines

Quoted 3 lines

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps.

I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands? This question should be involving your series too.

Quoted 16 lines

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

How about failing if the provider name is not unique unless user gives the actual binary path?

Quoted 27 lines · Masami Hiramatsu

Hi,

Quoted 24 lines · Hemant Kumar

Hi Namhyung,

Quoted 21 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 12 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

Quoted 8 lines

Quoted 3 lines

Quoted 3 lines

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps.

??

The ~/.debug/.build-id/<build-id> (actually first 2 hexdigits are used for directory name) is a symlink to a cached binary.

  $ file .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b 
  .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b: symbolic link to `../../usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b'

  $ file .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b 
  .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xc4a6080026388b02245390f8aae770c73b5d0e45, stripped

Hmm.. it seems the file utility prints the build-id as a sequence of 4 byte little-endian integers.

Quoted 4 lines

I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

I think a single .probe file is enough for this, no?

Quoted 21 lines

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands? This question should be involving your series too.

Quoted 16 lines

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

What I'm saying is for managing cache not the usage of the cached events. IIUC you keep hash entry for all events to find matching file, but I think it can be managed in provider level as events in a single provider will live in a single binary.

Btw, I think we should support such multiple events to like

  # perf record -e %provider_xxx:* -e %provider_yyy:prefix_*

Quoted 3 lines

How about failing if the provider name is not unique unless user gives the actual binary path?

It looks like a possible option. :)

Thanks,
Namhyung

[Off topic] I really don't like that the current SDT's semaphore. If the user apps see the instruction at the probe point, it is easy to check whether the event is enabled or not. Thus I recommend to change its implementation and update version instead of supporting current semaphore by perftools.

Quoted 4 lines · Masami Hiramatsu

[Off topic] I really don't like that the current SDT's semaphore. If the user apps see the instruction at the probe point, it is easy to check whether the event is enabled or not. Thus I recommend to change its implementation and update version instead of supporting current semaphore by perftools.

You and I have banged heads on this before, but I don't think checking the instruction is a simple as you seem to think. I invite you to prototype this, and if you get it working we can discuss the tradeoffs.

The good news is that other tools (stap and gdb) won't need to care. If the SDT semaphore goes automatic, then we can just set that note field to zero, unused from the tool's perspective.

Another tactic is to just discourage developers from using the semaphore in the first place, as it's a completely optional feature. The marker is just a NOP, so adding some "if (enabled) {...}" around it is often a useless load and branch. It does make sense if the probe wants to provide some expensively-computed arguments though, like cpython does to prepare a function name string. So if you see a project testing the semaphore around simple arguments, I'd suggest they just probe directly instead.

Quoted 43 lines · Namhyung Kim

Quoted 27 lines · Masami Hiramatsu

Hi,

Quoted 24 lines · Hemant Kumar

Hi Namhyung,

Quoted 21 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

   https://lkml.org/lkml/2014/10/31/207

Quoted 12 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps.

??

The ~/.debug/.build-id/<build-id> (actually first 2 hexdigits are used for directory name) is a symlink to a cached binary.

  $ file .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b 
  .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b: symbolic link to `../../usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b'

Quoted 6 lines

  $ file .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b 
  .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xc4a6080026388b02245390f8aae770c73b5d0e45, stripped

Hmm.. it seems the file utility prints the build-id as a sequence of 4 byte little-endian integers.

Ah, I saw the kernel's build-id file...

Quoted 5 lines

Quoted 4 lines

I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

I think a single .probe file is enough for this, no?

I hope to be so, but SDT is a bit different, that may have a semaphore. Currently we just ignore it. But in some application, SDT events never be hit without enabling the semaphores. I'm not sure how we can enable it. Systemtap enables it by modifying application memory(data) on the fly. Maybe perftools can be done it by using ptrace, but it's not for ftrace.

[Off topic] I really don't like that the current SDT's semaphore. If the user apps see the instruction at the probe point, it is easy to check whether the event is enabled or not. Thus I recommend to change its implementation and update version instead of supporting current semaphore by perftools.

Quoted 20 lines

Quoted 20 lines

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands? This question should be involving your series too.

Quoted 15 lines

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

Quoted 4 lines

What I'm saying is for managing cache not the usage of the cached events. IIUC you keep hash entry for all events to find matching file, but I think it can be managed in provider level as events in a single provider will live in a single binary.

Usually, the SDT provider names are managed by hand, as it is unique.

Quoted 4 lines

Btw, I think we should support such multiple events to like

  # perf record -e %provider_xxx:* -e %provider_yyy:prefix_*

what will be placed in the xxx and yyy ?

Thank you,

Quoted 7 lines

Quoted 2 lines

How about failing if the provider name is not unique unless user gives the actual binary path?

It looks like a possible option. :)

Thanks,
Namhyung

So, what should be our way forward here in case of SDT patchset wrt event_cache patchset? Shall we wait for event_cache patchset to be merged and then redesign the sdt_cache patchset according to new event_cache?
Or, we can go ahead with the current sdt patchset (implementing the latest review comments) and we can change the sdt_cache according to the new event_cache design as and when required?

What do you think?

The paths of buildid cache of binary/symbols are not changed.

Quoted 1 line · Masami Hiramatsu

The paths of buildid cache of binary/symbols are not changed.

Could we get an option in .perfconfig to disable the buildid cache? Its pointless diskspace wastage for some of us.

Quoted 5 lines · Masami Hiramatsu

Quoted 1 line

The paths of buildid cache of binary/symbols are not changed.

Could we get an option in .perfconfig to disable the buildid cache? Its pointless diskspace wastage for some of us.

I was going to respond with the same. This:

[buildid]
# Default, disable using /dev/null
dir = /dev/null

disables the build-id cache. Hopefully that extends to all the other caches getting discussed.

Hello,

Here, I've tried to describe my idea of perf-cache subcommand interface. It is just a design review, not implemented yet :)
Please give me your comments/ideas!

Command-line Synopsis
=====================

Current "perf buildid-cache [options]" are directly mapped to "perf cache --buildid [options]".

And adding --sdt for managing SDT caches as below.

Add or update SDT events in <FILES>

    perf cache --sdt --add|--update <FILES>
  Remove all SDT events for <FILES>
    perf cache --sdt --remove <FILES>

  List all SDT events
    perf cache --sdt --list

And --probes for managing probe-caches as below.

Add new probe-cache entries for kernel, <PATH> or <MOD>.

    perf cache --probe [--exec <PATH>|--module <MOD>] --add <SPEC>

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.

    perf cache --probe --del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Or remove all entires for given FILES

    perf cache --probe --remove <FILES>

List the probe caches(including SDT) for kernel, <PATH>, or/and <BUILDID>.

    perf cache --probe --list [@<PATH>][#<BUILDID>]

Query the probe definitions.

    perf cache --probe --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Note that --probes also can be used for managing SDT events, which has % prefix e.g.
Add all SDT events for <PATH>

    perf cache --probe --exec <PATH> --add '%*:*'

Remove some SDT events for <PATH>

    perf cache --probe --del '%some:events@<PATH>'

Or remove all SDT events for <BUILDID>

    perf cache --probe --del '%*:*#<BUILDID>'

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

This SDT/probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#path:PATH
p:%PROVIDER/EVENT PATH:OFFSET [ARGS]

p:PROBE/EVENT _text+OFFSET [ARGS]
----

Normal probes and SDT cache entries can be mixed in a cache file, we'll load all the entries and filter by % prefixes.

Thank you,

Quoted 10 lines

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

Quoted 5 lines

This SDT/probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#path:PATH
p:%PROVIDER/EVENT PATH:OFFSET [ARGS]

I think this format isn't accepted by the uprobe_events file (because of the '%') if we want to cat and grep into uprobe_events file directly. Although, in case of perf record, we can process it to remove '%' and then write into uprobe_events file. And, '%' is required as a flag for the SDT entries.

So, either we can modify the uprobe_events file to ignore '%' in the event group name (with an additional patch in the SDT series) or we can leave it to the user to write a script which can do some processing to remove '%' and then echo it to uprobe_events file.
What would you think?

Hello,

Here, I've tried to describe my idea of perf-cache subcommand interface. It is just a design review, not implemented yet :)
Please give me your comments/ideas!

Command-line Synopsis
=====================

Current "perf buildid-cache [options]" are directly mapped to "perf cache --buildid [options]".

And adding --sdt for managing SDT caches as below.

Add or update SDT events in <FILES>

    perf cache --sdt --add|--update <FILES>

Remove all SDT events for <FILES>

    perf cache --sdt --remove <FILES>

  List all SDT events
    perf cache --sdt --list

And --probes for managing probe-caches as below.

Add new probe-cache entries for kernel, <PATH> or <MOD>.

    perf cache --probe [--exec <PATH>|--module <MOD>] --add <SPEC>

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.

    perf cache --probe --del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Or remove all entires for given FILES

    perf cache --probe --remove <FILES>

List the probe caches(including SDT) for kernel, <PATH>, or/and <BUILDID>.

    perf cache --probe --list [@<PATH>][#<BUILDID>]

Query the probe definitions.

    perf cache --probe --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Note that --probes also can be used for managing SDT events, which has % prefix e.g.
Add all SDT events for <PATH>

    perf cache --probe --exec <PATH> --add '%*:*'

Remove some SDT events for <PATH>

    perf cache --probe --del '%some:events@<PATH>'

Or remove all SDT events for <BUILDID>

    perf cache --probe --del '%*:*#<BUILDID>'

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid

and that is linked to ~/.debug/.probes/.buildid/bu/ildid

# To avoid conflict with files under /probes/*, I picked up .probes/.

This SDT/probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#path:PATH
p:%PROVIDER/EVENT PATH:OFFSET [ARGS]
p:PROBE/EVENT _text+OFFSET [ARGS]
----

Normal probes and SDT cache entries can be mixed in a cache file, we'll load all the entries and filter by % prefixes.

Em Fri, Nov 07, 2014 at 05:21:08PM +0900, Masami Hiramatsu escreveu:

Quoted 12 lines

Hello,

Here, I've tried to describe my idea of perf-cache subcommand interface. It is just a design review, not implemented yet :)
Please give me your comments/ideas!

Command-line Synopsis
=====================

Current "perf buildid-cache [options]" are directly mapped to "perf cache --buildid [options]".

Quoted 1 line

And adding --sdt for managing SDT caches as below.

Can't we avoid having to specify the content? I.e. the tool can surely be smart enough to figure that out, no?

We should, as much as possible, to make things more compact yet unambiguous, IMHO.

Quoted 3 lines

Add or update SDT events in <FILES>

    perf cache --sdt --add|--update <FILES>

Can automagically figure this out, so, it would become just:

	Add or update events in <FILES>

	  perf cache --add/--update <FILES>

Quoted 3 lines

Remove all SDT events for <FILES>

    perf cache --sdt --remove <FILES>

Ditto.

Quoted 2 lines

  List all SDT events
    perf cache --sdt --list

That is ok, since the cache can hold different types of contents, but:

	perf cache --list

should show everything, with some marking showing the content type.

Quoted 5 lines

And --probes for managing probe-caches as below.

Add new probe-cache entries for kernel, <PATH> or <MOD>.

    perf cache --probe [--exec <PATH>|--module <MOD>] --add <SPEC>

No need to specify --probe

Quoted 2 lines

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.

    perf cache --probe --del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Ditto

Quoted 2 lines

Or remove all entires for given FILES

    perf cache --probe --remove <FILES>

Ok

Quoted 2 lines

List the probe caches(including SDT) for kernel, <PATH>, or/and <BUILDID>.

    perf cache --probe --list [@<PATH>][#<BUILDID>]

Can figure out the kind by the initial character, so no need to specify --probe

Quoted 2 lines

Query the probe definitions.

    perf cache --probe --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Ditto

Quoted 4 lines

Note that --probes also can be used for managing SDT events, which has % prefix e.g.
Add all SDT events for <PATH>

    perf cache --probe --exec <PATH> --add '%*:*'

See? --probe is smart enough to deal with SDT and probe caches, it can disambiguate by looking at the first char.

Quoted 3 lines

Remove some SDT events for <PATH>

    perf cache --probe --del '%some:events@<PATH>'

No need for --probe

Quoted 2 lines

Or remove all SDT events for <BUILDID>

    perf cache --probe --del '%*:*#<BUILDID>'

Ditto

Quoted 7 lines

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid

Here I think we could group everything related to a /path/to/bin into:

~/.debug/path/to/bin/bu/ild/

With one file per content type:

~/.debug/path/to/bin/bu/ild/elf
~/.debug/path/to/bin/bu/ild/probe

That leaves room for us to add more file formats (CTF anyone? The Dtrace one, Compact C Type Information, I mean).

So that if we wanted to pick everything related to a pathname we could do:

tar cf gcc.debug.tar ~/.debug/usr/bin/gcc/

If we instead wanted to pick all files to a specific gcc build id, we would do:

tar cf gcc.debug.tar ~/.debug/usr/bin/gcc/bu/ild/

Quoted 1 line

and that is linked to ~/.debug/.probes/.buildid/bu/ildid

Since this would break compatibility with the existing cache format, we could as well allow collecting everything related to a build id, by making the link be also in this fashion:

tar cf bu.ildid.tar ~/.debug/.build-id/bu/ildid/

Because we would have:

~/.debug/.build-id/bu/ildid/elf -> ~/.debug/usr/bin/gcc/bu/ild/elf ~/.debug/.build-id/bu/ildid/probe -> ~/.debug/usr/bin/gcc/bu/ild/probe

And also links to the files that have no pathnames:

~/.debug/.build-id/bu/ildid/vdso -> ~/.debug/[vdso]/bu/ild ~/.debug/.build-id/bu/ildid/kallsyms -> ~/.debug/[kernel.kallsyms]/bu/ild ~/.debug/.build-id/bu/ildid/kcore -> ~/.debug/[kcore]/bu/ild

Hello,

Here, I've tried to describe my idea of perf-cache subcommand interface. It is just a design review, not implemented yet :)
Please give me your comments/ideas!

Command-line Synopsis
=====================

Current "perf buildid-cache [options]" are directly mapped to "perf cache --buildid [options]".

And adding --sdt for managing SDT caches as below.

Add or update SDT events in <FILES>

    perf cache --sdt --add|--update <FILES>

Remove all SDT events for <FILES>

    perf cache --sdt --remove <FILES>

  List all SDT events
    perf cache --sdt --list

And --probes for managing probe-caches as below.

Add new probe-cache entries for kernel, <PATH> or <MOD>.

    perf cache --probe [--exec <PATH>|--module <MOD>] --add <SPEC>

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.

    perf cache --probe --del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Or remove all entires for given FILES

    perf cache --probe --remove <FILES>

List the probe caches(including SDT) for kernel, <PATH>, or/and <BUILDID>.

    perf cache --probe --list [@<PATH>][#<BUILDID>]

Query the probe definitions.

    perf cache --probe --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Note that --probes also can be used for managing SDT events, which has % prefix e.g.
Add all SDT events for <PATH>

    perf cache --probe --exec <PATH> --add '%*:*'

Remove some SDT events for <PATH>

    perf cache --probe --del '%some:events@<PATH>'

Or remove all SDT events for <BUILDID>

    perf cache --probe --del '%*:*#<BUILDID>'

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

This SDT/probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#path:PATH
p:%PROVIDER/EVENT PATH:OFFSET [ARGS]

p:PROBE/EVENT _text+OFFSET [ARGS]
----

Normal probes and SDT cache entries can be mixed in a cache file, we'll load all the entries and filter by % prefixes.

Thank you,

Hi Masami,

2014-11-07 (금), 17:21 +0900, Masami Hiramatsu:

Quoted 45 lines

Hello,

Here, I've tried to describe my idea of perf-cache subcommand interface. It is just a design review, not implemented yet :)
Please give me your comments/ideas!

Command-line Synopsis
=====================

Current "perf buildid-cache [options]" are directly mapped to "perf cache --buildid [options]".

And adding --sdt for managing SDT caches as below.

Add or update SDT events in <FILES>

    perf cache --sdt --add|--update <FILES>

Remove all SDT events for <FILES>

    perf cache --sdt --remove <FILES>

  List all SDT events
    perf cache --sdt --list

And --probes for managing probe-caches as below.

Add new probe-cache entries for kernel, <PATH> or <MOD>.

    perf cache --probe [--exec <PATH>|--module <MOD>] --add <SPEC>

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.

    perf cache --probe --del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Or remove all entires for given FILES

    perf cache --probe --remove <FILES>

List the probe caches(including SDT) for kernel, <PATH>, or/and <BUILDID>.

    perf cache --probe --list [@<PATH>][#<BUILDID>]

Query the probe definitions.

    perf cache --probe --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Note that --probes also can be used for managing SDT events, which has % prefix e.g.
Add all SDT events for <PATH>

    perf cache --probe --exec <PATH> --add '%*:*'

Quoted 5 lines

Remove some SDT events for <PATH>

    perf cache --probe --del '%some:events@<PATH>'

Or remove all SDT events for <BUILDID>

    perf cache --probe --del '%*:*#<BUILDID>'

I'd prefer sub-command to option for this mandatory (and exclusive) behavior. What about like this?

perf cache kprobe add [-m <module>] <spec> [<spec>...] perf cache kprobe del [<group>:]<event> [<event>...] perf cache kprobe list [-m <module>]

perf cache uprobe add -x <path> <spec> [<spec>...] perf cache uprobe del [<group>:]<event> [<event>...] perf cache uprobe list [-x <path>]

perf cache sdt [add|del|update] <file> [<file>...] perf cache sdt list [-b <build-id>] [<file>...]

Quoted 10 lines

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

However, to be used with perf record, we need a way to find a matching probe cache file from a event name (or group/provider name, preferably). What about having something like below:

  $ cat ~/.debug/.probes/event.map
  PROVIDER1 /path/to/some/where
  PROVIDER2 /path/to/other/place

When perf record see a cached event used, it first searches its provider/group name from the event.map file. And then read bulid-id from the matching file on the path and finally find a cached event definition from ~/.debug/.probes/.buildid/bu/ildid file.

Quoted 6 lines

This SDT/probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#path:PATH
p:%PROVIDER/EVENT PATH:OFFSET [ARGS]

It seems PATH is redundant and we don't need to repeat it everyline IMHO. Since it need post-processing anyway, maybe it's better to just make it simpler, like:

%:PROVIDER/EVENT OFFSET [ARGS]

Hello,

Here is the second try for the probe-cache. This version simplifies the synopsis, and unifies the SDT and probe caches. Please give me your comments/ideas!

Command-line Synopsis
=====================

Add elf(or symbols) and probe-caches of SDT if exists in <FILES>
perf cache --add <FILES> [--probe <SPEC>] # for user programs

perf cache --kcore <FILE> [--probe <SPEC>] # for kcore ?

perf cache --probe <SPEC> # for the current kernel

Remove caches related to <FILES> or <BUILDIDS>
perf cache --remove <FILES>|<BUILDIDS>

Show all probe caches(including SDT) or buildids
perf cache --list [probe|buildid]

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.
perf cache　--probe-del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Query the probe definitions.
perf cache --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

File Format
===========
All the cache files are placed under ~/.debug/ by default.
Elf caches are ~/.debug/path/to/file/bu/ildid/elf
Symbols caches are ~/.debug/path/to/file/bu/ildid/syms Probes(and SDT) are ~/.debug/path/to/file/bu/ildid/probes
And ~/.debug/path/to/file/bu/ildid/ is linked to ~/.debug/.buildid/bu/ildid Optionally, we can gzip the probes file.

This probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#spec:SDT
p:sdt_<PROVIDER>/<EVENT> PATH:OFFSET [ARGS]
...
#spec:* $params
p:probe_<GROUP>/<EVENT> _text+OFFSET [ARGS]
...
----
So the #spec: line gives the information what probe spec has been given. This will be used for updating.

And all the "probe_" and "sdt_" prefix will be replaced by % in the command line, e.g.

perf record -e %<PROVIDER>/<EVENT> -> this records sdt_PROVIDER/EVENT

Thank you,

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Em Mon, Nov 10, 2014 at 07:59:24PM +0900, Masami Hiramatsu escreveu:

Quoted 9 lines

Hello,

Here is the second try for the probe-cache. This version simplifies the synopsis, and unifies the SDT and probe caches. Please give me your comments/ideas!

Command-line Synopsis
=====================

Quoted 2 lines

Add elf(or symbols) and probe-caches of SDT if exists in <FILES>
perf cache --add <FILES> [--probe <SPEC>] # for user programs

Why the --probe above? Shouldn't this be just (if you are talking about ELF files only):

perf cache --add <FILES>

Quoted 1 line

perf cache --kcore <FILE> [--probe <SPEC>] # for kcore ?

Adrian, aren't kcore files easily identifiable as such and thus could be added as:

perf cache --add <FILES>

Quoted 1 line

perf cache --probe <SPEC> # for the current kernel

Why do we need a --probe here? Don't they always start with a character that is seldomly used in ELF file names and thus we could get away with not requiring --probe?

Quoted 6 lines

Remove caches related to <FILES> or <BUILDIDS>
perf cache --remove <FILES>|<BUILDIDS>

Show all probe caches(including SDT) or buildids
perf cache --list [probe|buildid]

Quoted 2 lines

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.
perf cache　--probe-del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Ditto, i.e. can't we just use:

perf cache　--remove [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

And it figure out that this is a probe that is being removed?

Quoted 2 lines

Query the probe definitions.
perf cache --query [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Replace --query with --show?

Quoted 9 lines

File Format
===========
All the cache files are placed under ~/.debug/ by default.
Elf caches are ~/.debug/path/to/file/bu/ildid/elf
Symbols caches are ~/.debug/path/to/file/bu/ildid/syms Probes(and SDT) are ~/.debug/path/to/file/bu/ildid/probes
And ~/.debug/path/to/file/bu/ildid/ is linked to ~/.debug/.buildid/bu/ildid Optionally, we can gzip the probes file.

Ok!

Quoted 26 lines

This probe caches contain probe-definitions as following format.
----
#buildid:BUILDID
#spec:SDT
p:sdt_<PROVIDER>/<EVENT> PATH:OFFSET [ARGS]
...
#spec:* $params
p:probe_<GROUP>/<EVENT> _text+OFFSET [ARGS]
...
----
So the #spec: line gives the information what probe spec has been given. This will be used for updating.

And all the "probe_" and "sdt_" prefix will be replaced by % in the command line, e.g.

perf record -e %<PROVIDER>/<EVENT> -> this records sdt_PROVIDER/EVENT

Thank you,

Signature · 4 lines

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: [email protected]

Quoted 10 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 10, 2014 at 07:59:24PM +0900, Masami Hiramatsu escreveu:

Quoted 9 lines

Here is the second try for the probe-cache. This version simplifies the synopsis, and unifies the SDT and probe caches. Please give me your comments/ideas!

Command-line Synopsis
=====================

Add elf(or symbols) and probe-caches of SDT if exists in <FILES>
perf cache --add <FILES> [--probe <SPEC>] # for user programs

Quoted 4 lines

Why the --probe above? Shouldn't this be just (if you are talking about ELF files only):

perf cache --add <FILES>

Yes, for the elf and sdt cache, we don't need --probe.
Note that "[]" means optional. If we would like to add some probe cache, we need a spec of probe definition.

Quoted 8 lines

Quoted 1 line

perf cache --kcore <FILE> [--probe <SPEC>] # for kcore ?

Adrian, aren't kcore files easily identifiable as such and thus could be added as:

perf cache --add <FILES>

Quoted 1 line

perf cache --probe <SPEC> # for the current kernel

Quoted 3 lines

Why do we need a --probe here? Don't they always start with a character that is seldomly used in ELF file names and thus we could get away with not requiring --probe?

This is only for adding the probe cache (not elf, nor sdt), which requires a probe definition. Moreover, I'd like to unify the specification of the probe definition with perf-probe. In that case, --probe is more natural.

Quoted 11 lines

Quoted 8 lines

Remove caches related to <FILES> or <BUILDIDS>
perf cache --remove <FILES>|<BUILDIDS>

Show all probe caches(including SDT) or buildids
perf cache --list [probe|buildid]

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.
perf cache　--probe-del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Ditto, i.e. can't we just use:

Quoted 3 lines

perf cache　--remove [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

And it figure out that this is a probe that is being removed?

In most cases, it may be OK, but it is also possible to cause unexpected result when mis-typing. I think if <FILE> is always starting at '/', it is easy to identify.

Em Tue, Nov 11, 2014 at 03:53:42PM +0900, Masami Hiramatsu escreveu:

Quoted 12 lines

Quoted 10 lines · Arnaldo Carvalho de Melo

Em Mon, Nov 10, 2014 at 07:59:24PM +0900, Masami Hiramatsu escreveu:

Quoted 9 lines

Here is the second try for the probe-cache. This version simplifies the synopsis, and unifies the SDT and probe caches. Please give me your comments/ideas!

Command-line Synopsis
=====================

Add elf(or symbols) and probe-caches of SDT if exists in <FILES>
perf cache --add <FILES> [--probe <SPEC>] # for user programs

Quoted 8 lines

Quoted 4 lines

Why the --probe above? Shouldn't this be just (if you are talking about ELF files only):

perf cache --add <FILES>

Yes, for the elf and sdt cache, we don't need --probe.
Note that "[]" means optional. If we would like to add some probe cache, we need a spec of probe definition.

I understand that, its just that it looked superfluous at that specific place, where you are explaining how to add ELF files.

Quoted 9 lines

Quoted 8 lines

Quoted 1 line

perf cache --kcore <FILE> [--probe <SPEC>] # for kcore ?

Adrian, aren't kcore files easily identifiable as such and thus could be added as:

perf cache --add <FILES>

Quoted 1 line

perf cache --probe <SPEC> # for the current kernel

Quoted 7 lines

Quoted 3 lines

Why do we need a --probe here? Don't they always start with a character that is seldomly used in ELF file names and thus we could get away with not requiring --probe?

This is only for adding the probe cache (not elf, nor sdt), which requires a probe definition. Moreover, I'd like to unify the specification of the probe definition with perf-probe. In that case, --probe is more natural.

What I meant was, what is wrong with replacing:

perf cache --probe <SPEC> # for the current kernel

With:

perf cache --add <PROBE-SPEC> # for the current kernel

And have it figure out that what is being added is a probe and do the right thing?

Quoted 15 lines

Quoted 14 lines

Quoted 8 lines

Remove caches related to <FILES> or <BUILDIDS>
perf cache --remove <FILES>|<BUILDIDS>

Show all probe caches(including SDT) or buildids
perf cache --list [probe|buildid]

Delete existing probe-cache entries for kernel, <PATH> or/and <BUILDID>.
perf cache　--probe-del [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

Ditto, i.e. can't we just use:

perf cache　--remove [<GROUP>:]<EVENT>[@<PATH>][#<BUILDID>]

And it figure out that this is a probe that is being removed?

Quoted 3 lines

In most cases, it may be OK, but it is also possible to cause unexpected result when mis-typing. I think if <FILE> is always starting at '/', it is easy to identify.

We can keep the explicit switch (--probe-del) perhaps to resolve ambiguities, if they happen, but make it so that it is not strictly required for the common case.

Quoted 10 lines · Arnaldo Carvalho de Melo

What I meant was, what is wrong with replacing:

perf cache --probe <SPEC> # for the current kernel

With:

perf cache --add <PROBE-SPEC> # for the current kernel

And have it figure out that what is being added is a probe and do the right thing?

As I've said previously, PROBE-SPEC can be same as FILES (imagine that a binary file which has same name function in the kernel.)
Moreover, PROBE-SPEC requires the target binary(or kernel module) except for kernel probes. In that case, anyway we need -x or -m options with file-path for --add, that is very strange.

e.g.

For me,

perf cache --add ./binary --probe '*'

looks more natural than

perf cache --add '*' -exec ./binary

since in other cases(sdt/elf), we'll just do

perf cache --add ./binary

Hi Masami,

Quoted 30 lines · Masami Hiramatsu

Quoted 10 lines · Arnaldo Carvalho de Melo

What I meant was, what is wrong with replacing:

perf cache --probe <SPEC> # for the current kernel

With:

perf cache --add <PROBE-SPEC> # for the current kernel

And have it figure out that what is being added is a probe and do the right thing?

As I've said previously, PROBE-SPEC can be same as FILES (imagine that a binary file which has same name function in the kernel.)
Moreover, PROBE-SPEC requires the target binary(or kernel module) except for kernel probes. In that case, anyway we need -x or -m options with file-path for --add, that is very strange.

e.g.

For me,

perf cache --add ./binary --probe '*'

looks more natural than

perf cache --add '*' -exec ./binary

since in other cases(sdt/elf), we'll just do

Quoted 1 line

perf cache --add ./binary

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

Quoted 39 lines · Namhyung Kim

Hi Masami,

Quoted 24 lines · Masami Hiramatsu

Quoted 10 lines · Arnaldo Carvalho de Melo

What I meant was, what is wrong with replacing:

perf cache --probe <SPEC> # for the current kernel

With:

perf cache --add <PROBE-SPEC> # for the current kernel

And have it figure out that what is being added is a probe and do the right thing?

As I've said previously, PROBE-SPEC can be same as FILES (imagine that a binary file which has same name function in the kernel.)
Moreover, PROBE-SPEC requires the target binary(or kernel module) except for kernel probes. In that case, anyway we need -x or -m options with file-path for --add, that is very strange.

e.g.

For me,

perf cache --add ./binary --probe '*'

Quoted 7 lines

looks more natural than

perf cache --add '*' -exec ./binary

since in other cases(sdt/elf), we'll just do

perf cache --add ./binary

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

Quoted 1 line

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

OK, that's good to me. And I think --elf/--sdt is meaningless. Only --probe option is required, since we can scan the elf file to add sdt cache when adding elf binary :)

Quoted 10 lines · Masami Hiramatsu

Quoted 5 lines · Namhyung Kim

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

OK, that's good to me. And I think --elf/--sdt is meaningless. Only --probe option is required, since we can scan the elf file to add sdt cache when adding elf binary :)

BTW, what should we do if we put the probe cache on current running kernel?

perf cache add --probe <probe-spec>

and have no binary argument, is it OK?

Quoted 12 lines · Masami Hiramatsu

Quoted 10 lines · Masami Hiramatsu

Quoted 5 lines · Namhyung Kim

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

OK, that's good to me. And I think --elf/--sdt is meaningless. Only --probe option is required, since we can scan the elf file to add sdt cache when adding elf binary :)

Quoted 5 lines

BTW, what should we do if we put the probe cache on current running kernel?

perf cache add --probe <probe-spec>

and have no binary argument, is it OK?

Hmm.. what about passing /proc/kallsyms and/or /boot/vmlinux for that?

Now I found that we need to special-case the kallsyms non-ELF file :)

Quoted 5 lines · Namhyung Kim

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

OK, that's good to me. And I think --elf/--sdt is meaningless.

Only --probe option is required, since we can scan the elf file to add sdt cache when adding elf binary :)

Hi Masami,

Quoted 8 lines · Masami Hiramatsu

Quoted 5 lines · Namhyung Kim

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

OK, that's good to me. And I think --elf/--sdt is meaningless.

Maybe not :)

I'm considering the opposite side - by providing the options, we also support the negative ones too. So --no-elf and/or --no-sdt options are possible. Also the positive options can be used with del(ete) subcommand to remove some contents selectively.

I think it'd be helpful as we sometimes don't want to do that for some reason. For example, current perf record adds binary (elf) files to the cache automatically iff it's accessed. But what about SDTs? Should we add SDTs at the same time? If not, what if we try to add existing elf files only for SDTs?

Quoted 29 lines

Quoted 10 lines · Arnaldo Carvalho de Melo

What I meant was, what is wrong with replacing:

perf cache --probe <SPEC> # for the current kernel

With:

perf cache --add <PROBE-SPEC> # for the current kernel

And have it figure out that what is being added is a probe and do the right thing?

As I've said previously, PROBE-SPEC can be same as FILES (imagine that a binary file which has same name function in the kernel.)
Moreover, PROBE-SPEC requires the target binary(or kernel module) except for kernel probes. In that case, anyway we need -x or -m options with file-path for --add, that is very strange.

e.g.

For me,

perf cache --add ./binary --probe '*'

looks more natural than

perf cache --add '*' -exec ./binary

since in other cases(sdt/elf), we'll just do

Quoted 1 line

perf cache --add ./binary

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

Em Mon, Nov 17, 2014 at 12:08:59PM +0900, Namhyung Kim escreveu:

Quoted 5 lines

I prefer this too. But I'd like make the 'add' part a subcommand rather than option like we do in perf kmem/kvm/list/lock/mem/sched ... And it can handle multiple files at once. What about this?

perf cache add [--elf|--sdt|--probe <spec>] <binary> [<binary>...]

In the end we can have it both ways, i.e. if the user does just:

perf cache add something

or:

    perf cache add --elf something

And 'something' is an ELF file, then in the first case (no --elf specified) it will figure it out (checking the magic number, etc) and do the right thing.

In the second case since we're being more verbose and think we know what 'something' is (an ELF file) the tool can check if it indeed is an ELF file and if not, bail out.

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

Quoted 5 lines · Masami Hiramatsu

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

Quoted 3 lines

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

A little bit late: but why ~/.debug? Why not $XDG_CACHE_HOME/perf/ as

the root for all perf related files? debug is not unique nor is it sufficient

to meet file hierarchy: man file-hierarchy(8)

Quoted 5 lines · Masami Hiramatsu

File Format
===========
All the cache files are placed under ~/.debug/ by default. The paths of buildid cache of binary/symbols are not changed.

Quoted 3 lines

The SDT/probe caches are placed under the ~/.debug/.probes/path/to/bin/bu/ildid and that is linked to ~/.debug/.probes/.buildid/bu/ildid
# To avoid conflict with files under /probes/*, I picked up .probes/.

A little bit late: but why ~/.debug? Why not $XDG_CACHE_HOME/perf/ as the root for all perf related files? debug is not unique nor is it sufficient to meet file hierarchy: man file-hierarchy(8)

# env | grep XDG | wc -l
0

Which renders it useless crap in my book.

I agree on the .debug name being somewhat generic, also, it would be:
man 8 file-hierarchy, but that too fails:

$ man file-hierarchy
No manual entry for file-hierarchy

Quoted 27 lines · Masami Hiramatsu

Hi,

Quoted 24 lines · Hemant Kumar

Hi Namhyung,

Quoted 21 lines · Namhyung Kim

Hi Hemant,

As you know, you need to keep an eye on how (kprobes) event cache patchset from Masami settles down. For those who aren't CC'ed, please see the link below:

    https://lkml.org/lkml/2014/10/31/207

Quoted 12 lines · Hemant Kumar

This patch adds support to perf to record SDT events. When invoked, the SDT event is looked up in the sdt-cache. If its found, an entry is made silently to uprobe_events file and then recording is invoked, and then the entry for the SDT event in uprobe_events is silently discarded.

The SDT events are already stored in a cache file (/var/cache/perf/perf-sdt-file.cache).
Although the file_hash table helps in addition or deletion of SDT events from the cache, its not of much use when it comes to probing the actual SDT event, because the key to this hash list is a file name and not the SDT event name (which is given as an argument to perf record). So, we won't be able to hash into it.

Quoted 7 lines

Quoted 3 lines

Quoted 3 lines

It likely to be ended up with per-file or per-buildid cache files under ~/.debug directory. In this case we also need to have the (central) event-to-cache table anyway IMHO.

What we are talking is to make a new caching file with buildid under .debug/.
We already has ~/.debug/.build-id/<build-id> for string the binary symbol maps.

??

The ~/.debug/.build-id/<build-id> (actually first 2 hexdigits are used for directory name) is a symlink to a cached binary.

   $ file .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b
   .debug/.build-id/00/08a6c4028b3826f8905324c770e7aa450e5d3b: symbolic link to `../../usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b'

   $ file .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b
   .debug/usr/lib64/libgtk-3.so.0.400.4/0008a6c4028b3826f8905324c770e7aa450e5d3b: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=0xc4a6080026388b02245390f8aae770c73b5d0e45, stripped

Hmm.. it seems the file utility prints the build-id as a sequence of 4 byte little-endian integers.

Quoted 4 lines

I think there are 2 options, one is expanding the current build-id file format to include sdt and probe-event caches. The other is to add ~/.debug/.build-id/<build-id>.probe and ~/.debug/.build-id/<build-id>.sdt for caching probe/sdt information.

I think a single .probe file is enough for this, no?

Quoted 20 lines

And also, user interface is a discussion point. This series defines new sdt-cache command, and we already have buildid-cache command. We should have probe-cache command too? or consolidate those cache managing commands? This question should be involving your series too.

Quoted 15 lines

Quoted 10 lines

Quoted 6 lines

To avoid this problem, we can create another hash list "event_hash" list which will be maintained along with the file_hash list. Whenever a user invokes 'perf record -e %provider:event, perf should initialize the event_hash list and the file_hash list.
The key to event_hash list is calculated from the event name and its provider name.

Isn't it enough just to use provide name? I guess the provider names are (should be?) unique among a system although there's no absolute guarantee for that.

Yes, there is no guarantee for the provider names to be unique. If we use only provider name with "perf record", then, what if a user wants to trace
only a specific SDT event (not all the events for that provider)? What do you think?

What I'm saying is for managing cache not the usage of the cached events. IIUC you keep hash entry for all events to find matching file, but I think it can be managed in provider level as events in a single provider will live in a single binary.

Btw, I think we should support such multiple events to like

   # perf record -e %provider_xxx:* -e %provider_yyy:prefix_*

Quoted 2 lines

How about failing if the provider name is not unique unless user gives the actual binary path?

It looks like a possible option. :)