Skip to contentSkip to conversation

perf tools: add meta-data header support in pipe mode

PATCH In review Stephane Eranian 6 participants 35 messages May 15, 2012 – May 24, 2012 perf events

lore
Share & tools
lore
#0 lore raw

v2 cover perf tools: add meta-data header support in pipe mode

This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -

We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.

In pipe mode, there is no meta-data header structures at the beginning of the streamed data given we cannot seek in a pipe. Instead, we need to create pseudo record types for each of the possible features, e.g., hostname, cpuid, event_desc, and so on. Those pseudo records are guaranteed to be before any actual sample records, therefore perf report/annotate are guaranteed to get the information they need before processing the first sample.

The series also fixes perf inject to actually inject build-ids and buildid-list to work better with pipe mode, i.e., print the build-ids.

With this series:

 $ perf record -o - noploop 2 | perf inject -b | perf report -i -
      # ========
      # captured on: Fri Jan 20 18:13:55 2012
      # ========
      #
      # hostname : quad
      # os release : 3.2.0-rc7-tip
      # perf version : 3.2.0
      # arch : x86_64
      # nrcpus online : 4
      # nrcpus avail : 4
      # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
      # cpuid : GenuineIntel,6,15,11
      # total memory : 8092884 kB
      ...
      # HEADER_CPU_TOPOLOGY info available, use -I to display
      noploop for 2 seconds
      [ perf record: Woken up 1 times to write data ]
      [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
        99.80%  noploop  noploop            [.] noploop
         0.19%  noploop  [kernel.kallsyms]  [k] radix_tree_gang_lookup

V2 is just a rebase to 3.4.0-rc7.

Stephane Eranian (6):
perf inject: fix broken perf inject -b
perf tools: fix piped mode read code
perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA perf record: add meta-data support for pipe-mode
perf: make perf buildid-list work better with pipe mode

Signature · 1 line
1.7.4.1
#1 lore raw

PATCH v2 1/5 perf inject: fix broken perf inject -b

perf inject -b was broken. It would not inject any build_id into the stream. Furthermore, it would strip samples from the stream.

The reason was a missing initialization of the event attribute structure. The perf_tool.tool.attr() callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.

The patch simply adds back the call to perf_event__process_attr() to initialize the evlist structure and now build_ids are again injected.

---
 tools/perf/builtin-inject.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
tools/perf/builtin-inject.c +5 -0
index 09c1061..3beab48 100644
@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,
 static int perf_event__repipe_attr(union perf_event *event,
 				   struct perf_evlist **pevlist __used)
 {
+	int ret;
+	ret = perf_event__process_attr(event, pevlist);
+	if (ret)
+		return ret;
+
 	return perf_event__repipe_synth(NULL, event, NULL);
 }
 
Signature · 1 line
1.7.4.1
#2 lore raw

Re: [PATCH v2 1/5] perf inject: fix broken perf inject -b

Quoted 24 lines · Stephane Eranian

perf inject -b was broken. It would not inject any build_id into the stream. Furthermore, it would strip samples from the stream.

The reason was a missing initialization of the event attribute structure. The perf_tool.tool.attr() callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.

The patch simply adds back the call to perf_event__process_attr() to initialize the evlist structure and now build_ids are again injected.

---
  tools/perf/builtin-inject.c |    5 +++++
  1 files changed, 5 insertions(+), 0 deletions(-)
tools/perf/builtin-inject.c +0 -0
index 09c1061..3beab48 100644

tools/perf/builtin-inject.c static int perf_event__repipe_tracing_data_synth(union perf_event *event, · line 61

Quoted 2 lines
tools/perf/builtin-inject.c +0 -0
@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,
  static int perf_event__repipe_attr(union perf_event *event,
  				   struct perf_evlist **pevlist __used)

The _used can be dropped now that pevlist is used.

tools/perf/builtin-inject.c static int perf_event__repipe_tracing_data_synth(union perf_event *event, · line 70

Quoted 9 lines
tools/perf/builtin-inject.c +5 -0
  {
+	int ret;
+	ret = perf_event__process_attr(event, pevlist);
+	if (ret)
+		return ret;
+
  	return perf_event__repipe_synth(NULL, event, NULL);
  }
 

The change seems right, but I could not recreate the dropped events. In both cases running:

perf record -o - -m 256 /tmp/a.sh | perf inject -b | perf report -i - -D

where /tmp/a.sh is:

#!/bin/sh
cd /opt/sw/ahern/kernels/kernel.git
make O=/tmp/kbuild -j 32 >/dev/null 2>&1

just to give it some "real work."

With your patch:

           TOTAL events:     284211
            MMAP events:      69236
            COMM events:      10863
            EXIT events:      22858
        THROTTLE events:         11
      UNTHROTTLE events:         11
            FORK events:      11428
          SAMPLE events:     169779
            ATTR events:          1
      EVENT_TYPE events:          1
        BUILD_ID events:         23
cycles stats:
           TOTAL events:     284186
            MMAP events:      69236
            COMM events:      10863
            EXIT events:      22858
        THROTTLE events:         11
      UNTHROTTLE events:         11
            FORK events:      11428
          SAMPLE events:     169779

Current acme/core:

           TOTAL events:     288990
            MMAP events:      69236
            COMM events:      10863
            EXIT events:      22858
        THROTTLE events:         20
      UNTHROTTLE events:         20
            FORK events:      11428
          SAMPLE events:     174538
            ATTR events:          1
      EVENT_TYPE events:          1
        BUILD_ID events:         25
cycles stats:
           TOTAL events:     288963
            MMAP events:      69236
            COMM events:      10863
            EXIT events:      22858
        THROTTLE events:         20
      UNTHROTTLE events:         20
            FORK events:      11428
          SAMPLE events:     174538

David

[1] Both tests have Namhyung's patch:
https://lkml.org/lkml/2012/5/15/280

#3 lore raw

[tip:perf/core] perf inject: Fix broken perf inject -b

In reply to Stephane Eranian
Commit-ID:  1a1ed1ba6778a5bc5702cebe276ab080a0b78115
Gitweb:     http://git.kernel.org/tip/1a1ed1ba6778a5bc5702cebe276ab080a0b78115
Author:     Stephane Eranian <[email protected]>
AuthorDate: Tue, 15 May 2012 13:28:11 +0200
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 22 May 2012 12:59:28 -0300

perf inject: Fix broken perf inject -b

perf inject -b was broken. It would not inject any build_id into the stream. Furthermore, it would strip samples from the stream.

The reason was a missing initialization of the event attribute structure. The perf_tool.tool.attr() callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.

The patch simply adds back the call to perf_event__process_attr() to initialize the evlist structure and now build_ids are again injected.

---
 tools/perf/builtin-inject.c |    5 +++++
 1 files changed, 5 insertions(+), 0 deletions(-)
tools/perf/builtin-inject.c +5 -0
index 09c1061..3beab48 100644
@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,
 static int perf_event__repipe_attr(union perf_event *event,
 				   struct perf_evlist **pevlist __used)
 {
+	int ret;
+	ret = perf_event__process_attr(event, pevlist);
+	if (ret)
+		return ret;
+
 	return perf_event__repipe_synth(NULL, event, NULL);
 }
 
#4 lore raw

PATCH v2 2/5 perf tools: fix piped mode read code

In reply to Stephane Eranian

In __perf_session__process_pipe_events(), there was a risk we would read more than what a union perf_event struct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.

In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union perf_event, yet it was used as the backing to the read in:

union perf_event event;
void *p;

  size = event->header.size;

  p = &event;
  p += sizeof(struct perf_event_header);
  if (size - sizeof(struct perf_event_header)) {
    err = readn(self->fd, p, size - sizeof(struct perf_event_header));

We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.

---
 tools/perf/util/session.c |   34 ++++++++++++++++++++++++++--------
 1 files changed, 26 insertions(+), 8 deletions(-)
tools/perf/util/session.c +26 -8
index 4dcc8f3..af6a5f0 100644
@@ -1064,8 +1064,9 @@ volatile int session_done;
 static int __perf_session__process_pipe_events(struct perf_session *self,
 					       struct perf_tool *tool)
 {
-	union perf_event event;
-	uint32_t size;
+	union perf_event *event;
+	uint32_t size, cur_size = 0;
+	void *buf = NULL;
 	int skip = 0;
 	u64 head;
 	int err;
@@ -1074,8 +1075,14 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 	perf_tool__fill_defaults(tool);
 
 	head = 0;
+	cur_size = sizeof(union perf_event);
+
+	buf = malloc(cur_size);
+	if (!buf)
+		return -errno;
 more:
-	err = readn(self->fd, &event, sizeof(struct perf_event_header));
+	event = buf;
+	err = readn(self->fd, event, sizeof(struct perf_event_header));
 	if (err <= 0) {
 		if (err == 0)
 			goto done;
@@ -1085,13 +1092,23 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 	}
 
 	if (self->header.needs_swap)
-		perf_event_header__bswap(&event.header);
+		perf_event_header__bswap(&event->header);
 
-	size = event.header.size;
+	size = event->header.size;
 	if (size == 0)
 		size = 8;
 
-	p = &event;
+	if (size > cur_size) {
+		void *new = realloc(buf, size);
+		if (!new) {
+			pr_err("failed to allocate memory to read event\n");
+			goto out_err;
+		}
+		buf = new;
+		cur_size = size;
+		event = buf;
+	}
+	p = event;
 	p += sizeof(struct perf_event_header);
 
 	if (size - sizeof(struct perf_event_header)) {
@@ -1107,9 +1124,9 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 		}
 	}
 
-	if ((skip = perf_session__process_event(self, &event, tool, head)) < 0) {
+	if ((skip = perf_session__process_event(self, event, tool, head)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
-		       head, event.header.size, event.header.type);
+		       head, event->header.size, event->header.type);
 		err = -EINVAL;
 		goto out_err;
 	}
@@ -1124,6 +1141,7 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 done:
 	err = 0;
 out_err:
+	free(buf);
 	perf_session__warn_about_errors(self, tool);
 	perf_session_free_sample_buffers(self);
 	return err;
Signature · 1 line
1.7.4.1
#5 lore raw

Re: [PATCH v2 2/5] perf tools: fix piped mode read code

Quoted 27 lines · Stephane Eranian

In __perf_session__process_pipe_events(), there was a risk we would read more than what a union perf_event struct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.

In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union perf_event, yet it was used as the backing to the read in:

   union perf_event event;
   void *p;

   size = event->header.size;

   p =&event;
   p += sizeof(struct perf_event_header);
   if (size - sizeof(struct perf_event_header)) {
     err = readn(self->fd, p, size - sizeof(struct perf_event_header));

We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.

Quoted 5 lines
---
  tools/perf/util/session.c |   34 ++++++++++++++++++++++++++--------
  1 files changed, 26 insertions(+), 8 deletions(-)

I don't have a file/perf with different sized events, but the change makes sense.

Reviewed-and-tested-by David Ahern <[email protected]>
#6 lore raw

[tip:perf/core] perf tools: Fix piped mode read code

In reply to Stephane Eranian
Commit-ID:  444d28663936e286c752f05feca44d6041b1fce4
Gitweb:     http://git.kernel.org/tip/444d28663936e286c752f05feca44d6041b1fce4
Author:     Stephane Eranian <[email protected]>
AuthorDate: Tue, 15 May 2012 13:28:12 +0200
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 22 May 2012 12:59:52 -0300

perf tools: Fix piped mode read code

In __perf_session__process_pipe_events(), there was a risk we would read more than what a union perf_event struct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.

In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union perf_event, yet it was used as the backing to the read in:

union perf_event event;
void *p;

  size = event->header.size;

  p = &event;
  p += sizeof(struct perf_event_header);
  if (size - sizeof(struct perf_event_header)) {
    err = readn(self->fd, p, size - sizeof(struct perf_event_header));

We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.

---
 tools/perf/util/session.c |   34 ++++++++++++++++++++++++++--------
 1 files changed, 26 insertions(+), 8 deletions(-)
tools/perf/util/session.c +26 -8
index 17c9ace..93d355d 100644
@@ -1098,8 +1098,9 @@ volatile int session_done;
 static int __perf_session__process_pipe_events(struct perf_session *self,
 					       struct perf_tool *tool)
 {
-	union perf_event event;
-	uint32_t size;
+	union perf_event *event;
+	uint32_t size, cur_size = 0;
+	void *buf = NULL;
 	int skip = 0;
 	u64 head;
 	int err;
@@ -1108,8 +1109,14 @@ static int __perf_session__process_pipe_events(struct perf_session *self,
 	perf_tool__fill_defaults(tool);
 
 	head = 0;
+	cur_size = sizeof(union perf_event);
+
+	buf = malloc(cur_size);
+	if (!buf)
+		return -errno;
 more:
-	err = readn(self->fd, &event, sizeof(struct perf_event_header));
+	event = buf;
+	err = readn(self->fd, event, sizeof(struct perf_event_header));
 	if (err <= 0) {
 		if (err == 0)
 			goto done;
@@ -1119,13 +1126,23 @@ more:
 	}
 
 	if (self->header.needs_swap)
-		perf_event_header__bswap(&event.header);
+		perf_event_header__bswap(&event->header);
 
-	size = event.header.size;
+	size = event->header.size;
 	if (size == 0)
 		size = 8;
 
-	p = &event;
+	if (size > cur_size) {
+		void *new = realloc(buf, size);
+		if (!new) {
+			pr_err("failed to allocate memory to read event\n");
+			goto out_err;
+		}
+		buf = new;
+		cur_size = size;
+		event = buf;
+	}
+	p = event;
 	p += sizeof(struct perf_event_header);
 
 	if (size - sizeof(struct perf_event_header)) {
@@ -1141,9 +1158,9 @@ more:
 		}
 	}
 
-	if ((skip = perf_session__process_event(self, &event, tool, head)) < 0) {
+	if ((skip = perf_session__process_event(self, event, tool, head)) < 0) {
 		pr_err("%#" PRIx64 " [%#x]: failed to process type: %d\n",
-		       head, event.header.size, event.header.type);
+		       head, event->header.size, event->header.type);
 		err = -EINVAL;
 		goto out_err;
 	}
@@ -1158,6 +1175,7 @@ more:
 done:
 	err = 0;
 out_err:
+	free(buf);
 	perf_session__warn_about_errors(self, tool);
 	perf_session_free_sample_buffers(self);
 	return err;
#7 lore raw

PATCH v2 3/5 perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA

In reply to Stephane Eranian

To match the PERF_RECORD_HEADER_TRACING_DATA record type. This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.

---
 tools/perf/builtin-record.c |    4 ++--
 tools/perf/util/header.c    |   10 +++++-----
 tools/perf/util/header.h    |    2 +-
 3 files changed, 8 insertions(+), 8 deletions(-)
tools/perf/builtin-record.c +2 -2
index d19058a..21cf20b 100644
@@ -396,7 +396,7 @@ static void perf_record__mmap_read_all(struct perf_record *rec)
 			perf_record__mmap_read(rec, &rec->evlist->mmap[i]);
 	}
 
-	if (perf_header__has_feat(&rec->session->header, HEADER_TRACE_INFO))
+	if (perf_header__has_feat(&rec->session->header, HEADER_TRACING_DATA))
 		write_output(rec, &finished_round_event, sizeof(finished_round_event));
 }
 
@@ -478,7 +478,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 		perf_header__clear_feat(&session->header, HEADER_BUILD_ID);
 
 	if (!have_tracepoints(&evsel_list->entries))
-		perf_header__clear_feat(&session->header, HEADER_TRACE_INFO);
+		perf_header__clear_feat(&session->header, HEADER_TRACING_DATA);
 
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
tools/perf/util/header.c +5 -5
index 5385980..2dd5edf 100644
@@ -437,7 +437,7 @@ static bool perf_session__read_build_ids(struct perf_session *session, bool with
 	return ret;
 }
 
-static int write_trace_info(int fd, struct perf_header *h __used,
+static int write_tracing_data(int fd, struct perf_header *h __used,
 			    struct perf_evlist *evlist)
 {
 	return read_tracing_data(fd, &evlist->entries);
@@ -1472,7 +1472,7 @@ static int perf_header__read_build_ids(struct perf_header *header,
 	return err;
 }
 
-static int process_trace_info(struct perf_file_section *section __unused,
+static int process_tracing_data(struct perf_file_section *section __unused,
 			      struct perf_header *ph __unused,
 			      int feat __unused, int fd)
 {
@@ -1508,11 +1508,11 @@ struct feature_ops {
 		.full_only = true }
 
 /* feature_ops not implemented: */
-#define print_trace_info		NULL
-#define print_build_id			NULL
+#define print_tracing_data	NULL
+#define print_build_id		NULL
 
 static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
-	FEAT_OPP(HEADER_TRACE_INFO,	trace_info),
+	FEAT_OPP(HEADER_TRACING_DATA,	tracing_data),
 	FEAT_OPP(HEADER_BUILD_ID,	build_id),
 	FEAT_OPA(HEADER_HOSTNAME,	hostname),
 	FEAT_OPA(HEADER_OSRELEASE,	osrelease),
tools/perf/util/header.h +1 -1
index 21a6be0..2d42b3e 100644
@@ -12,7 +12,7 @@
 enum {
 	HEADER_RESERVED		= 0,	/* always cleared */
 	HEADER_FIRST_FEATURE	= 1,
-	HEADER_TRACE_INFO	= 1,
+	HEADER_TRACING_DATA	= 1,
 	HEADER_BUILD_ID,
 
 	HEADER_HOSTNAME,
Signature · 1 line
1.7.4.1
Reviewed-by on v2 3/5
#8 lore raw

Re: [PATCH v2 3/5] perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA

Quoted 6 lines · Stephane Eranian

To match the PERF_RECORD_HEADER_TRACING_DATA record type. This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.

#9 lore raw

[tip:perf/core] perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA

In reply to Stephane Eranian
Commit-ID:  2eeaaa095d155d47d47d5df07571105b8ae76ffd
Gitweb:     http://git.kernel.org/tip/2eeaaa095d155d47d47d5df07571105b8ae76ffd
Author:     Stephane Eranian <[email protected]>
AuthorDate: Tue, 15 May 2012 13:28:13 +0200
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 22 May 2012 12:57:46 -0300

perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA

To match the PERF_RECORD_HEADER_TRACING_DATA record type.

This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.

---
 tools/perf/builtin-record.c |    4 ++--
 tools/perf/util/header.c    |   10 +++++-----
 tools/perf/util/header.h    |    2 +-
 3 files changed, 8 insertions(+), 8 deletions(-)
tools/perf/builtin-record.c +2 -2
index 8a3dfac..9c76add 100644
@@ -396,7 +396,7 @@ static void perf_record__mmap_read_all(struct perf_record *rec)
 			perf_record__mmap_read(rec, &rec->evlist->mmap[i]);
 	}
 
-	if (perf_header__has_feat(&rec->session->header, HEADER_TRACE_INFO))
+	if (perf_header__has_feat(&rec->session->header, HEADER_TRACING_DATA))
 		write_output(rec, &finished_round_event, sizeof(finished_round_event));
 }
 
@@ -478,7 +478,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 		perf_header__clear_feat(&session->header, HEADER_BUILD_ID);
 
 	if (!have_tracepoints(&evsel_list->entries))
-		perf_header__clear_feat(&session->header, HEADER_TRACE_INFO);
+		perf_header__clear_feat(&session->header, HEADER_TRACING_DATA);
 
 	if (!rec->opts.branch_stack)
 		perf_header__clear_feat(&session->header, HEADER_BRANCH_STACK);
tools/perf/util/header.c +5 -5
index 5385980..2dd5edf 100644
@@ -437,7 +437,7 @@ static bool perf_session__read_build_ids(struct perf_session *session, bool with
 	return ret;
 }
 
-static int write_trace_info(int fd, struct perf_header *h __used,
+static int write_tracing_data(int fd, struct perf_header *h __used,
 			    struct perf_evlist *evlist)
 {
 	return read_tracing_data(fd, &evlist->entries);
@@ -1472,7 +1472,7 @@ out:
 	return err;
 }
 
-static int process_trace_info(struct perf_file_section *section __unused,
+static int process_tracing_data(struct perf_file_section *section __unused,
 			      struct perf_header *ph __unused,
 			      int feat __unused, int fd)
 {
@@ -1508,11 +1508,11 @@ struct feature_ops {
 		.full_only = true }
 
 /* feature_ops not implemented: */
-#define print_trace_info		NULL
-#define print_build_id			NULL
+#define print_tracing_data	NULL
+#define print_build_id		NULL
 
 static const struct feature_ops feat_ops[HEADER_LAST_FEATURE] = {
-	FEAT_OPP(HEADER_TRACE_INFO,	trace_info),
+	FEAT_OPP(HEADER_TRACING_DATA,	tracing_data),
 	FEAT_OPP(HEADER_BUILD_ID,	build_id),
 	FEAT_OPA(HEADER_HOSTNAME,	hostname),
 	FEAT_OPA(HEADER_OSRELEASE,	osrelease),
tools/perf/util/header.h +1 -1
index 21a6be0..2d42b3e 100644
@@ -12,7 +12,7 @@
 enum {
 	HEADER_RESERVED		= 0,	/* always cleared */
 	HEADER_FIRST_FEATURE	= 1,
-	HEADER_TRACE_INFO	= 1,
+	HEADER_TRACING_DATA	= 1,
 	HEADER_BUILD_ID,
 
 	HEADER_HOSTNAME,
#10 lore raw

PATCH v2 4/5 perf record: add meta-data support for pipe-mode

In reply to Stephane Eranian

This patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -

Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.

The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for BUILD_ID and TRACING_DATA.

We have modified util/header.c such that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct called feat_fd which encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.

It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis. That is always the case with the example shown below. However, one could also do:

$ perf record -o - noploop 2 | perf inject -b >perf.data
$ cat perf.data | perf report -i -

With this patch, it is possible to get:

 $ perf record -o - noploop 2 | perf inject -b | perf report -i -
      # ========
      # captured on: Fri Jan 20 18:13:55 2012
      # ========
      #
      # hostname : quad
      # os release : 3.2.0-rc7-tip
      # perf version : 3.2.0
      # arch : x86_64
      # nrcpus online : 4
      # nrcpus avail : 4
      # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
      # cpuid : GenuineIntel,6,15,11
      # total memory : 8092884 kB
      ...
      # HEADER_CPU_TOPOLOGY info available, use -I to display
      noploop for 2 seconds
      [ perf record: Woken up 1 times to write data ]
      [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
        99.80%  noploop  noploop            [.] noploop
         0.19%  noploop  [kernel.kallsyms]  [k] radix_tree_gang_lookup
---
 tools/perf/builtin-annotate.c |    5 +
 tools/perf/builtin-inject.c   |    1 +
 tools/perf/builtin-record.c   |    7 +
 tools/perf/builtin-report.c   |    7 +-
 tools/perf/util/event.c       |   12 +
 tools/perf/util/event.h       |   18 ++
 tools/perf/util/header.c      |  532 +++++++++++++++++++++++++++++------------
 tools/perf/util/header.h      |    9 +
 tools/perf/util/session.c     |   17 ++-
 tools/perf/util/tool.h        |    4 +-
 10 files changed, 449 insertions(+), 163 deletions(-)
tools/perf/builtin-annotate.c +5 -0
index 806e0a2..fe42298 100644
@@ -247,6 +247,11 @@ int cmd_annotate(int argc, const char **argv, const char *prefix __used)
 			.mmap	= perf_event__process_mmap,
 			.comm	= perf_event__process_comm,
 			.fork	= perf_event__process_task,
+			.attr	= perf_event__process_attr,
+			.event_type	 = perf_event__process_event_type,
+			.tracing_data	 = perf_event__process_tracing_data,
+			.build_id	 = perf_event__process_build_id,
+			.feature	 = perf_event__process_feature,
 			.ordered_samples = true,
 			.ordering_requires_timestamps = true,
 		},
tools/perf/builtin-inject.c +1 -0
index 3beab48..65b0529 100644
@@ -217,6 +217,7 @@ struct perf_tool perf_inject = {
 	.event_type	= perf_event__repipe_event_type_synth,
 	.tracing_data	= perf_event__repipe_tracing_data_synth,
 	.build_id	= perf_event__repipe_op2_synth,
+	.feature	= perf_event__repipe_op2_synth,
 };
 
 extern volatile int session_done;
tools/perf/builtin-record.c +7 -0
index 21cf20b..e2c5310 100644
@@ -531,6 +531,13 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)
 	}
 
 	if (opts->pipe_output) {
+		err = perf_event__synthesize_features(tool, session, evsel_list,
+						   process_synthesized_event);
+		if (err < 0) {
+			pr_err("Couldn't synthesize features.\n");
+			return err;
+		}
+
 		err = perf_event__synthesize_attrs(tool, session,
 						   process_synthesized_event);
 		if (err < 0) {
tools/perf/builtin-report.c +4 -3
index d58e414..be5d544 100644
@@ -43,7 +43,6 @@ struct perf_report {
 	bool			force, use_tui, use_gtk, use_stdio;
 	bool			hide_unresolved;
 	bool			dont_use_callchains;
-	bool			show_full_info;
 	bool			show_threads;
 	bool			inverted_callchain;
 	struct perf_read_values	show_threads_values;
@@ -358,7 +357,8 @@ static int __cmd_report(struct perf_report *rep)
 	}
 
 	if (use_browser <= 0)
-		perf_session__fprintf_info(session, stdout, rep->show_full_info);
+		perf_session__fprintf_info(session, stdout,
+					   rep->tool.show_all_features);
 
 	if (rep->show_threads)
 		perf_read_values_init(&rep->show_threads_values);
@@ -562,6 +562,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __used)
 			.event_type	 = perf_event__process_event_type,
 			.tracing_data	 = perf_event__process_tracing_data,
 			.build_id	 = perf_event__process_build_id,
+			.feature	 = perf_event__process_feature,
 			.ordered_samples = true,
 			.ordering_requires_timestamps = true,
 		},
@@ -625,7 +626,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __used)
 		    "Look for files with symbols relative to this directory"),
 	OPT_STRING('C', "cpu", &report.cpu_list, "cpu",
 		   "list of cpus to profile"),
-	OPT_BOOLEAN('I', "show-info", &report.show_full_info,
+	OPT_BOOLEAN('I', "show-info", &report.tool.show_all_features,
 		    "Display extended information about perf.data file"),
 	OPT_BOOLEAN(0, "source", &symbol_conf.annotate_src,
 		    "Interleave source code with assembly code (default)"),
tools/perf/util/event.c +12 -0
index 2a6f33c..da3551f 100644
@@ -23,6 +23,18 @@ static const char *perf_event__names[] = {
 	[PERF_RECORD_HEADER_TRACING_DATA]	= "TRACING_DATA",
 	[PERF_RECORD_HEADER_BUILD_ID]		= "BUILD_ID",
 	[PERF_RECORD_FINISHED_ROUND]		= "FINISHED_ROUND",
+	[PERF_RECORD_HEADER_HOSTNAME]		= "HOSTNAME",
+	[PERF_RECORD_HEADER_OSRELEASE]		= "OSRELEASE",
+	[PERF_RECORD_HEADER_VERSION]		= "VERSION",
+	[PERF_RECORD_HEADER_ARCH]		= "ARCH",
+	[PERF_RECORD_HEADER_NRCPUS]		= "NRCPUS",
+	[PERF_RECORD_HEADER_CPUDESC]		= "CPUDESC",
+	[PERF_RECORD_HEADER_CPUID]		= "CPUID",
+	[PERF_RECORD_HEADER_TOTAL_MEM]		= "TOTAL_MEM",
+	[PERF_RECORD_HEADER_CMDLINE]		= "CMDLINE",
+	[PERF_RECORD_HEADER_EVENT_DESC]		= "EVENT_DESC",
+	[PERF_RECORD_HEADER_CPU_TOPOLOGY]	= "CPU_TOPOLOGY",
+	[PERF_RECORD_HEADER_NUMA_TOPOLOGY]	= "NUMA_TOPOLOGY",
 };
 
 const char *perf_event__name(unsigned int id)
tools/perf/util/event.h +18 -0
index 1b19728..e44aaec 100644
@@ -100,6 +100,18 @@ enum perf_user_event_type { /* above any possible kernel type */
 	PERF_RECORD_HEADER_TRACING_DATA		= 66,
 	PERF_RECORD_HEADER_BUILD_ID		= 67,
 	PERF_RECORD_FINISHED_ROUND		= 68,
+	PERF_RECORD_HEADER_HOSTNAME		= 69,
+	PERF_RECORD_HEADER_OSRELEASE		= 70,
+	PERF_RECORD_HEADER_VERSION		= 71,
+	PERF_RECORD_HEADER_ARCH			= 72,
+	PERF_RECORD_HEADER_NRCPUS		= 73,
+	PERF_RECORD_HEADER_CPUDESC		= 74,
+	PERF_RECORD_HEADER_CPUID		= 75,
+	PERF_RECORD_HEADER_TOTAL_MEM		= 76,
+	PERF_RECORD_HEADER_CMDLINE		= 77,
+	PERF_RECORD_HEADER_EVENT_DESC		= 78,
+	PERF_RECORD_HEADER_CPU_TOPOLOGY		= 79,
+	PERF_RECORD_HEADER_NUMA_TOPOLOGY	= 80,
 	PERF_RECORD_HEADER_MAX
 };
 
@@ -126,6 +138,11 @@ struct tracing_data_event {
 	u32 size;
 };
 
+struct feature_event {
+	struct perf_event_header header;
+	char data[]; /* size bytes of raw data specific to the feature */
+};
+
 union perf_event {
 	struct perf_event_header	header;
 	struct ip_event			ip;
@@ -139,6 +156,7 @@ union perf_event {
 	struct event_type_event		event_type;
 	struct tracing_data_event	tracing_data;
 	struct build_id_event		build_id;
+	struct feature_event		feat;
 };
 
 void perf_event__print_totals(void);
tools/perf/util/header.c +375 -157

1,076 lines not on this page · raw message

tools/perf/util/header.h +9 -0
index 2d42b3e..af4db9e 100644
@@ -98,6 +98,15 @@ int build_id_cache__add_s(const char *sbuild_id, const char *debugdir,
 			  const char *name, bool is_kallsyms);
 int build_id_cache__remove_s(const char *sbuild_id, const char *debugdir);
 
+int perf_event__synthesize_features(struct perf_tool *tool,
+				    struct perf_session *session,
+				    struct perf_evlist *evlist,
+				    perf_event__handler_t process);
+
+int perf_event__process_feature(struct perf_tool *tool,
+				union perf_event *event,
+				 struct perf_session *session);
+
 int perf_event__synthesize_attr(struct perf_tool *tool,
 				struct perf_event_attr *attr, u16 ids, u64 *id,
 				perf_event__handler_t process);
tools/perf/util/session.c +15 -2
index af6a5f0..8e5d74d 100644
@@ -390,6 +390,14 @@ static int process_event_type_stub(struct perf_tool *tool __used,
 	return 0;
 }
 
+static int process_feature_stub(struct perf_tool *tool __used,
+				union perf_event *event __used,
+				struct perf_session *session __used)
+{
+	dump_printf(": unhandled!\n");
+	return 0;
+}
+
 static int process_finished_round(struct perf_tool *tool,
 				  union perf_event *event,
 				  struct perf_session *session);
@@ -428,6 +436,8 @@ static void perf_tool__fill_defaults(struct perf_tool *tool)
 		else
 			tool->finished_round = process_finished_round_stub;
 	}
+	if (tool->feature == NULL)
+		tool->feature = process_feature_stub;
 }
 
 void mem_bswap_64(void *src, int byte_size)
@@ -530,8 +540,8 @@ static perf_event__swap_op perf_event__swap_ops[] = {
 	[PERF_RECORD_SAMPLE]		  = perf_event__all64_swap,
 	[PERF_RECORD_HEADER_ATTR]	  = perf_event__hdr_attr_swap,
 	[PERF_RECORD_HEADER_EVENT_TYPE]	  = perf_event__event_type_swap,
-	[PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
 	[PERF_RECORD_HEADER_BUILD_ID]	  = NULL,
+	[PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap,
 	[PERF_RECORD_HEADER_MAX]	  = NULL,
 };
 
@@ -947,6 +957,8 @@ static int perf_session__process_user_event(struct perf_session *session, union
 		return tool->build_id(tool, event, session);
 	case PERF_RECORD_FINISHED_ROUND:
 		return tool->finished_round(tool, event, session);
+	case PERF_RECORD_HEADER_HOSTNAME ... PERF_RECORD_HEADER_NUMA_TOPOLOGY:
+		return tool->feature(tool, event, session);
 	default:
 		return -EINVAL;
 	}
@@ -1509,6 +1521,7 @@ void perf_session__fprintf_info(struct perf_session *session, FILE *fp,
 
 	fprintf(fp, "# ========\n");
 	fprintf(fp, "# captured on: %s", ctime(&st.st_ctime));
-	perf_header__fprintf_info(session, fp, full);
 	fprintf(fp, "# ========\n#\n");
+	if (!session->fd_pipe)
+		perf_header__fprintf_info(session, fp, full);
 }
tools/perf/util/tool.h +3 -1
index b0e1aad..af16af1f 100644
@@ -42,9 +42,11 @@ struct perf_tool {
 	event_synth_op	tracing_data;
 	event_simple_op	event_type;
 	event_op2	finished_round,
-			build_id;
+			build_id,
+			feature;
 	bool		ordered_samples;
 	bool		ordering_requires_timestamps;
+	bool		show_all_features;
 };
 
 #endif /* __PERF_TOOL_H */
Signature · 1 line
1.7.4.1
#11 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 52 lines · Stephane Eranian

This patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -

Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.

The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for BUILD_ID and TRACING_DATA.

We have modified util/header.c such that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct called feat_fd which encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.

It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis. That is always the case with the example shown below. However, one could also do:

$ perf record -o - noploop 2 | perf inject -b>perf.data
$ cat perf.data | perf report -i -

With this patch, it is possible to get:

  $ perf record -o - noploop 2 | perf inject -b | perf report -i -
       # ========
       # captured on: Fri Jan 20 18:13:55 2012
       # ========
       #
       # hostname : quad
       # os release : 3.2.0-rc7-tip
       # perf version : 3.2.0
       # arch : x86_64
       # nrcpus online : 4
       # nrcpus avail : 4
       # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
       # cpuid : GenuineIntel,6,15,11
       # total memory : 8092884 kB
       ...
       # HEADER_CPU_TOPOLOGY info available, use -I to display
       noploop for 2 seconds
       [ perf record: Woken up 1 times to write data ]
       [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
         99.80%  noploop  noploop            [.] noploop
          0.19%  noploop  [kernel.kallsyms]  [k] radix_tree_gang_lookup
Quoted 1 line

I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...

David

#12 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 54 lines · David Ahern
Quoted 53 lines · Stephane Eranian

This patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -

Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.

The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for BUILD_ID and TRACING_DATA.

We have modified util/header.c such that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct called feat_fd which encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.

It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis.  That is always the case with the example shown below. However, one could also do:

$ perf record -o - noploop 2 | perf inject -b>perf.data
$ cat perf.data | perf report -i -

With this patch, it is possible to get:

 $ perf record -o - noploop 2 | perf inject -b | perf report -i -
      # ========
      # captured on: Fri Jan 20 18:13:55 2012
      # ========
      #
      # hostname : quad
      # os release : 3.2.0-rc7-tip
      # perf version : 3.2.0
      # arch : x86_64
      # nrcpus online : 4
      # nrcpus avail : 4
      # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
      # cpuid : GenuineIntel,6,15,11
      # total memory : 8092884 kB
      ...
      # HEADER_CPU_TOPOLOGY info available, use -I to display
      noploop for 2 seconds
      [ perf record: Woken up 1 times to write data ]
      [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
        99.80%  noploop  noploop            [.] noploop
         0.19%  noploop  [kernel.kallsyms]  [k] radix_tree_gang_lookup
Quoted 9 lines
Quoted 1 line

I love the feature and it works nicely, but there has been push back on adding more synthesized events.  Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...

Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.

Quoted 1 line

David

#13 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 7 lines · Stephane Eranian
Quoted 6 lines · David Ahern

I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...

Quoted 5 lines

Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.

Understood. I don't object to synthesized events.

Arnaldo: preferences?

David

#14 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Em Fri, May 18, 2012 at 10:50:45AM -0600, David Ahern escreveu:

Quoted 14 lines · Stephane Eranian
Quoted 7 lines · David Ahern
Quoted 6 lines

I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...

Quoted 5 lines

Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.

Quoted 3 lines

Understood. I don't object to synthesized events.

Arnaldo: preferences?

PeterZ was the one objecting to adding more userspace only events, Peter?

- Arnaldo

maintainer
#15 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 1 line · Arnaldo Carvalho de Melo

PeterZ was the one objecting to adding more userspace only events,

Ah, yes I was ;-)

So uhm the argument was something like perf_event_type is a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.

Furthermore most (if not all) the userspace thingies were setup (like in this case single session meta-data) things. So they don't belong in the event stream at all.

Quoted 1 line

From what I can remember all this is somehow related to how data is

passed to scripts or so and since there's only a single stdin everything is stuffed over it.

For the data file we should simply create another section in the header; and I think that is how it works these days.

For the script muck, I've really no idea how all that works, but why can't you wrap the stuff in another layer; have a header up front that says the next N bytes are meta-data and after that there's the regular data stream.

Or take some inspiration from one of the many multi-stream stream formats out there like mpeg or ogg or whatever.

#16 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 10 lines · Peter Zijlstra
Quoted 1 line · Arnaldo Carvalho de Melo

PeterZ was the one objecting to adding more userspace only events,

Ah, yes I was ;-)

So uhm the argument was something like perf_event_type is a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.

Yes, I did not start this but I understand why it was done that way.

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

So we need some namespace, so we can tag certain records as "synthetic" vs. kernel-generated. The way it was done is to assume that the kernel is not going to have more than 63 type of record types. That's where we run the risk of overlap one day. But for now we are at 10 kernel record types.

As such you need to inject certain header infos in the stream of kernel

Quoted 18 lines

Furthermore most (if not all) the userspace thingies were setup (like in this case single session meta-data) things. So they don't belong in the event stream at all.

From what I can remember all this is somehow related to how data is passed to scripts or so and since there's only a single stdin everything is stuffed over it.

For the data file we should simply create another section in the header; and I think that is how it works these days.

For the script muck, I've really no idea how all that works, but why can't you wrap the stuff in another layer; have a header up front that says the next N bytes are meta-data and after that there's the regular data stream.

Or take some inspiration from one of the many multi-stream stream formats out there like mpeg or ogg or whatever.

#17 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Hi,

Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 1 line · Arnaldo Carvalho de Melo

PeterZ was the one objecting to adding more userspace only events,

Ah, yes I was ;-)

So uhm the argument was something like perf_event_type is a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.

Quoted 9 lines

Yes, I did not start this but I understand why it was done that way.

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

IIUC, to add build-ids we should check all dso's whether hits or not, but it's not same as the case of meta-data since we can know it before processing. Am I missing something?

Thanks,
Namhyung

#18 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Em Wed, May 23, 2012 at 09:45:12AM +0900, Namhyung Kim escreveu:

Quoted 22 lines · Stephane Eranian
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Arnaldo Carvalho de Melo
Quoted 2 lines

PeterZ was the one objecting to adding more userspace only events,

Ah, yes I was ;-)

So uhm the argument was something like perf_event_type is a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.

Quoted 9 lines

Yes, I did not start this but I understand why it was done that way.

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

Quoted 3 lines

IIUC, to add build-ids we should check all dso's whether hits or not, but it's not same as the case of meta-data since we can know it before processing. Am I missing something?

To add build ids properly we should read it when the kernel loads a DSO, so that when PERF_RECORD_MMAP is sent to the event stream, the build id is there, so no need for post processing, etc.

The way we do it now is racy, long running sessions may experience DSO updating, etc.

I think we can figure out a way to avoid the seeks Stephane mentions so to get something done as Peter suggests.

But at the same time I don't think reusing the existing mechanism is such a big problem as Peter makes of it :-\

- Arnaldo

maintainer
#19 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 4 lines · Arnaldo Carvalho de Melo

But at the same time I don't think reusing the existing mechanism is such a big problem as Peter makes of it :-\

Probably not, but just know that I'll not take any such abuse into consideration when adding new record types.

That pipe mess is ill-designed and I don't care about it.

maintainer
#20 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

In reply to Stephane Eranian
Quoted 3 lines · Stephane Eranian

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode.

I know that, I said as much. But it is quite possible to multiplex streams without the need for seeking, look at these MKV containers that contain video, multiple-audio and multiple-subtitle tracks.

The fact is, whomever wrote that pipe mode took a bonghit before touching the keyboard. They didn't even ask if we could reserve a range for userspace events, let alone consider if it was a good idea to begin with.

If we ever manage to get splice() working you can't keep that pipe mode working anyway.

Pushing everything over a single stream isn't a scalable solution anyway in the same way that writing everything into the one file isn't the best thing and would ideally be replaced as well.

#21 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Stephane Eranian

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode.

Quoted 8 lines

I know that, I said as much. But it is quite possible to multiplex streams without the need for seeking, look at these MKV containers that contain video, multiple-audio and multiple-subtitle tracks.

The fact is, whomever wrote that pipe mode took a bonghit before touching the keyboard. They didn't even ask if we could reserve a range for userspace events, let alone consider if it was a good idea to begin with.

You're probably right, but the fact of the matter is, it's there now and we have lots of perf.data piling up all over the place. So you need to maintain compatibility with those. In my patch, I am just extending what's already there. I was not trying to rewrite the whole pipe mode + meta data logic (though I have done some of that already in the past).

Clearly pipe-mode was an after thought, but it is useful, I know people who use it. Obviously I use it, otherwise I would not necessarily have spent the time fixing the meta-data headers.

But I think, for the time being, my patch addresses an issue and does not prevent us for discussing a new file format for perf record, I have nothing against that.

Quoted 7 lines

If we ever manage to get splice() working you can't keep that pipe mode working anyway.

Pushing everything over a single stream isn't a scalable solution anyway in the same way that writing everything into the one file isn't the best thing and would ideally be replaced as well.

#22 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

In reply to Stephane Eranian
Quoted 6 lines · Stephane Eranian

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?

struct pipe_data {
	u32 length;
	u32 type;
	char data[0];		
}

It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.

David

#23 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:

Quoted 18 lines · Stephane Eranian
Quoted 6 lines

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?

struct pipe_data {
	u32 length;
	u32 type;
	char data[0];		
}
Quoted 4 lines

It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.

I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)

- Arnaldo

#24 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Quoted 24 lines · Arnaldo Carvalho de Melo

Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:

Quoted 18 lines · Stephane Eranian
Quoted 6 lines

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?

struct pipe_data {
	u32 length;
	u32 type;
	char data[0];		
}
Quoted 4 lines

It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.

Quoted 5 lines

I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)

- Arnaldo

I don't know about that 'easy' comment; that patch is fairly long. Hence my question about whether synthesized events are acceptable before spending time on it. ;-)

David

#25 lore raw

Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode

Em Thu, May 24, 2012 at 10:22:04AM -0600, David Ahern escreveu:

Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 24 lines

Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:

Quoted 22 lines · Stephane Eranian
Quoted 6 lines

The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.

Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?

struct pipe_data {
	u32 length;
	u32 type;
	char data[0];		
}

It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.

Quoted 5 lines
Quoted 3 lines

I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)

I don't know about that 'easy' comment; that patch is fairly long.

Easy in the sense that we didn't need to come up with any new mechanism to insert that meta data.

Quoted 2 lines

Hence my question about whether synthesized events are acceptable before spending time on it. ;-)

I think that we need to follow Peter suggestion, which looks to me similar to what you're suggesting, i.e. before the event stream, that will become multichannel anyway.

- Arnaldo

#26 lore raw

PATCH v2 5/5 perf: make perf buildid-list work better with pipe mode

In reply to Stephane Eranian

In order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.

$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H
noploop for 2 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.084 MB - (~3678 samples) ]
0000000000000000000000000000000000000000 [kernel.kallsyms]
3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploop

The reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream. The buildid for the kernel is provided by a BUILD_ID record. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately. In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.

So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.

---
 tools/perf/builtin-buildid-list.c |    6 +++++-
 tools/perf/util/build-id.c        |    2 ++
 2 files changed, 7 insertions(+), 1 deletions(-)
tools/perf/builtin-buildid-list.c +5 -1
index 5248046..6b2bcfb 100644
@@ -84,7 +84,11 @@ static int perf_session__list_build_ids(void)
 	if (filename__fprintf_build_id(session->filename, stdout))
 		goto out;
 
-	if (with_hits)
+	/*
+	 * in pipe-mode, the only way to get the buildids is to parse
+	 * the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID
+	 */
+	if (with_hits || session->fd_pipe)
 		perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 
 	perf_session__fprintf_dsos_buildid(session, stdout, with_hits);
tools/perf/util/build-id.c +2 -0
index dff9c7a..fd9a594 100644
@@ -65,6 +65,8 @@ struct perf_tool build_id__mark_dso_hit_ops = {
 	.mmap	= perf_event__process_mmap,
 	.fork	= perf_event__process_task,
 	.exit	= perf_event__exit_del_thread,
+	.attr		 = perf_event__process_attr,
+	.build_id	 = perf_event__process_build_id,
 };
 
 char *dso__build_id_filename(struct dso *self, char *bf, size_t size)
Signature · 1 line
1.7.4.1
#27 lore raw

Re: [PATCH v2 5/5] perf: make perf buildid-list work better with pipe mode

Quoted 19 lines · Stephane Eranian

In order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.

$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H
noploop for 2 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.084 MB - (~3678 samples) ]
0000000000000000000000000000000000000000 [kernel.kallsyms]
3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploop

The reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream. The buildid for the kernel is provided by a BUILD_ID record. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately. In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.

Quoted 8 lines

So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.

---
  tools/perf/builtin-buildid-list.c |    6 +++++-
  tools/perf/util/build-id.c        |    2 ++
  2 files changed, 7 insertions(+), 1 deletions(-)
Reviewed-and-tested-by David Ahern <[email protected]>
#28 lore raw

[tip:perf/core] perf buildid-list: Work better with pipe mode

In reply to Stephane Eranian
Commit-ID:  299c345208ae15fbc1e1c96dc1b6ac61e4e5da10
Gitweb:     http://git.kernel.org/tip/299c345208ae15fbc1e1c96dc1b6ac61e4e5da10
Author:     Stephane Eranian <[email protected]>
AuthorDate: Tue, 15 May 2012 13:28:15 +0200
Committer:  Arnaldo Carvalho de Melo <[email protected]>
CommitDate: Tue, 22 May 2012 13:03:54 -0300

perf buildid-list: Work better with pipe mode

In order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.

$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H
noploop for 2 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.084 MB - (~3678 samples) ]
0000000000000000000000000000000000000000 [kernel.kallsyms]
3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploop

The reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream.

The buildid for the kernel is provided by a BUILD_ID record. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately.

In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.

So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.

---
 tools/perf/builtin-buildid-list.c |    6 +++++-
 tools/perf/util/build-id.c        |    2 ++
 2 files changed, 7 insertions(+), 1 deletions(-)
tools/perf/builtin-buildid-list.c +5 -1
index 5248046..6b2bcfb 100644
@@ -84,7 +84,11 @@ static int perf_session__list_build_ids(void)
 	if (filename__fprintf_build_id(session->filename, stdout))
 		goto out;
 
-	if (with_hits)
+	/*
+	 * in pipe-mode, the only way to get the buildids is to parse
+	 * the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID
+	 */
+	if (with_hits || session->fd_pipe)
 		perf_session__process_events(session, &build_id__mark_dso_hit_ops);
 
 	perf_session__fprintf_dsos_buildid(session, stdout, with_hits);
tools/perf/util/build-id.c +2 -0
index dff9c7a..fd9a594 100644
@@ -65,6 +65,8 @@ struct perf_tool build_id__mark_dso_hit_ops = {
 	.mmap	= perf_event__process_mmap,
 	.fork	= perf_event__process_task,
 	.exit	= perf_event__exit_del_thread,
+	.attr		 = perf_event__process_attr,
+	.build_id	 = perf_event__process_build_id,
 };
 
 char *dso__build_id_filename(struct dso *self, char *bf, size_t size)
#29 lore raw
In reply to Stephane Eranian

Hi, Stephane

Quoted 8 lines · Stephane Eranian

This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -

We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.

In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).

Thanks,
Namhyung

#30 lore raw
Quoted 12 lines · Namhyung Kim

Hi, Stephane

Quoted 8 lines · Stephane Eranian

This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -

We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.

Quoted 7 lines

In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).

Thanks,
Namhyung

It is there -- see perf-report header. It gets dropped on pipe mode b/c the meta-data is in the header.

David

#31 lore raw

Hi,

Quoted 17 lines · David Ahern
Quoted 16 lines · Namhyung Kim

Hi, Stephane

Quoted 8 lines · Stephane Eranian

This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -

We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.

In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).

Quoted 8 lines
Quoted 3 lines

Thanks,
Namhyung

It is there -- see perf-report header. It gets dropped on pipe mode b/c the meta-data is in the header.

David

Do you mean this?

struct perf_report {
	struct perf_tool	tool;
	struct perf_session	*session;
	char const		*input_name;
	bool			force, use_tui, use_gtk, use_stdio;
	bool			hide_unresolved;
	bool			dont_use_callchains;
	bool			show_full_info;
	bool			show_threads;
	bool			inverted_callchain;
	struct perf_read_values	show_threads_values;
	const char		*pretty_printing_style;
	symbol_filter_t		annotate_init;
	const char		*cpu_list;
	const char		*symbol_filter_str;
	DECLARE_BITMAP(cpu_bitmap, MAX_NR_CPUS);
};

Thanks,
Namhyung

#32 lore raw

See tools/perf/util/header.{ch}
Runtime example:

perf script -i /tmp/perf.data -I

# ========

# captured on: Tue May 15 20:22:15 2012
# ========
#
# hostname : nxos-vdc-dev3
# os release : 3.4.0-rc7+
# perf version : 3.4.rc4.137.g978da3.dirty
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
# cpuid : GenuineIntel,6,26,5
# total memory : 24680324 kB
# cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop

...

'perf report --stdio' also shows it. It gets lost in the TUI.

David

#33 lore raw

Hi,

Quoted 20 lines · David Ahern

See tools/perf/util/header.{ch}
Runtime example:

perf script -i /tmp/perf.data -I

# ========

# captured on: Tue May 15 20:22:15 2012
# ========
#
# hostname : nxos-vdc-dev3
# os release : 3.4.0-rc7+
# perf version : 3.4.rc4.137.g978da3.dirty
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
# cpuid : GenuineIntel,6,26,5
# total memory : 24680324 kB
# cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop
Quoted 4 lines

...

'perf report --stdio' also shows it. It gets lost in the TUI.

Yeah, I know. What I mean is keeping the info in memory after parsing a perf.data file so that it can be used for displaying it on a dialog in perf report TUI (on demand) or checking arch before annotation or ...

Thanks,
Namhyung

#34 lore raw
In reply to David Ahern

Em Tue, May 15, 2012 at 08:38:05PM -0600, David Ahern escreveu:

Quoted 20 lines

See tools/perf/util/header.{ch}
Runtime example:

perf script -i /tmp/perf.data -I

# ========

# captured on: Tue May 15 20:22:15 2012
# ========
#
# hostname : nxos-vdc-dev3
# os release : 3.4.0-rc7+
# perf version : 3.4.rc4.137.g978da3.dirty
# arch : x86_64
# nrcpus online : 16
# nrcpus avail : 16
# cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz
# cpuid : GenuineIntel,6,26,5
# total memory : 24680324 kB
# cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop
Quoted 3 lines

...

'perf report --stdio' also shows it. It gets lost in the TUI.

Yeah, we need to add a splash screen with this info when the TUI starts and also ad a hotkey to show it at anytime.

- Arnaldo