perf tools: add meta-data header support in pipe mode
v2 cover perf tools: add meta-data header support in pipe mode
This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -
We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.
In pipe mode, there is no meta-data header structures at the beginning of the streamed data given we cannot seek in a pipe. Instead, we need to create pseudo record types for each of the possible features, e.g., hostname, cpuid, event_desc, and so on. Those pseudo records are guaranteed to be before any actual sample records, therefore perf report/annotate are guaranteed to get the information they need before processing the first sample.
The series also fixes perf inject to actually inject build-ids and buildid-list to work better with pipe mode, i.e., print the build-ids.
With this series:
$ perf record -o - noploop 2 | perf inject -b | perf report -i -
# ========
# captured on: Fri Jan 20 18:13:55 2012
# ========
#
# hostname : quad
# os release : 3.2.0-rc7-tip
# perf version : 3.2.0
# arch : x86_64
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
# cpuid : GenuineIntel,6,15,11
# total memory : 8092884 kB
...
# HEADER_CPU_TOPOLOGY info available, use -I to display
noploop for 2 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
99.80% noploop noploop [.] noploop
0.19% noploop [kernel.kallsyms] [k] radix_tree_gang_lookupV2 is just a rebase to 3.4.0-rc7.
Stephane Eranian (6):
perf inject: fix broken perf inject -b
perf tools: fix piped mode read code
perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA perf record: add meta-data support for pipe-mode
perf: make perf buildid-list work better with pipe mode
Signature · 1 line
1.7.4.1
PATCH v2 1/5 perf inject: fix broken perf inject -b
perf inject -b was broken. It would not inject any build_id into the stream. Furthermore, it would strip samples from the stream.
The reason was a missing initialization of the event attribute structure. The perf_tool.tool.attr() callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.
The patch simply adds back the call to perf_event__process_attr() to initialize the evlist structure and now build_ids are again injected.
--- tools/perf/builtin-inject.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
tools/perf/builtin-inject.c +5 -0
@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,static int perf_event__repipe_attr (union perf_event *event,struct perf_evlist **pevlist__used ) { +int ret; + ret =perf_event__process_attr (event, pevlist); +if (ret) +return ret; +return perf_event__repipe_synth (NULL , event,NULL ); }
Signature · 1 line
1.7.4.1
Re: [PATCH v2 1/5] perf inject: fix broken perf inject -b
Quoted 24 lines · Stephane Eranian
perf inject -b was broken. It would not inject any
build_idinto the stream. Furthermore, it would strip samples from the stream.The reason was a missing initialization of the event attribute structure. The
perf_tool.tool.attr()callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.The patch simply adds back the call to
perf_event__process_attr()to initialize the evlist structure and nowbuild_idsare again injected.Signed-off-by Stephane Eranian<[email protected]>--- tools/perf/builtin-inject.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)tools/perf/builtin-inject.c +0
-0
tools/perf/builtin-inject.c static int perf_event__repipe_tracing_data_synth(union perf_event *event, · line 61
Quoted 2 lines
tools/perf/builtin-inject.c +0
-0@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,static int perf_event__repipe_attr (union perf_event *event,struct perf_evlist **pevlist__used )
The _used can be dropped now that pevlist is used.
tools/perf/builtin-inject.c static int perf_event__repipe_tracing_data_synth(union perf_event *event, · line 70
Quoted 9 lines
tools/perf/builtin-inject.c +5
-0{ +int ret; + ret =perf_event__process_attr (event, pevlist); +if (ret) +return ret; +return perf_event__repipe_synth (NULL , event,NULL ); }
The change seems right, but I could not recreate the dropped events. In both cases running:
perf record -o - -m 256 /tmp/a.sh | perf inject -b | perf report -i - -D
where /tmp/a.sh is:
#!/bin/sh
cd /opt/sw/ahern/kernels/kernel.git
make O=/tmp/kbuild -j 32 >/dev/null 2>&1
just to give it some "real work."
With your patch:
TOTAL events: 284211
MMAP events: 69236
COMM events: 10863
EXIT events: 22858
THROTTLE events: 11
UNTHROTTLE events: 11
FORK events: 11428
SAMPLE events: 169779
ATTR events: 1
EVENT_TYPE events: 1
BUILD_ID events: 23
cycles stats:
TOTAL events: 284186
MMAP events: 69236
COMM events: 10863
EXIT events: 22858
THROTTLE events: 11
UNTHROTTLE events: 11
FORK events: 11428
SAMPLE events: 169779Current acme/core:
TOTAL events: 288990
MMAP events: 69236
COMM events: 10863
EXIT events: 22858
THROTTLE events: 20
UNTHROTTLE events: 20
FORK events: 11428
SAMPLE events: 174538
ATTR events: 1
EVENT_TYPE events: 1
BUILD_ID events: 25
cycles stats:
TOTAL events: 288963
MMAP events: 69236
COMM events: 10863
EXIT events: 22858
THROTTLE events: 20
UNTHROTTLE events: 20
FORK events: 11428
SAMPLE events: 174538David
[1] Both tests have Namhyung's patch:
https://lkml.org/lkml/2012/5/15/280
[tip:perf/core] perf inject: Fix broken perf inject -b
In reply to Stephane EranianCommit-ID: 1a1ed1ba6778a5bc5702cebe276ab080a0b78115 Gitweb: http://git.kernel.org/tip/1a1ed1ba6778a5bc5702cebe276ab080a0b78115 Author: Stephane Eranian <[email protected]> AuthorDate: Tue, 15 May 2012 13:28:11 +0200 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Tue, 22 May 2012 12:59:28 -0300
perf inject: Fix broken perf inject -b
perf inject -b was broken. It would not inject any build_id into the stream. Furthermore, it would strip samples from the stream.
The reason was a missing initialization of the event attribute structure. The perf_tool.tool.attr() callback was pointing to a simple repipe. But there was no initialization of the internal data structures to keep track of events and event ids. That later caused event id lookups to fail, and sample would get removed.
The patch simply adds back the call to perf_event__process_attr() to initialize the evlist structure and now build_ids are again injected.
--- tools/perf/builtin-inject.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
tools/perf/builtin-inject.c +5 -0
@@ -60,6 +60,11 @@ static int perf_event__repipe_tracing_data_synth(union perf_event *event,static int perf_event__repipe_attr (union perf_event *event,struct perf_evlist **pevlist__used ) { +int ret; + ret =perf_event__process_attr (event, pevlist); +if (ret) +return ret; +return perf_event__repipe_synth (NULL , event,NULL ); }
PATCH v2 2/5 perf tools: fix piped mode read code
In reply to Stephane EranianIn __perf_session__process_pipe_events(), there was a risk we would read more than what a union perf_event struct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.
In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union perf_event, yet it was used as the backing to the read in:
union perf_event event;
void *p;
size = event->header.size;
p = &event;
p += sizeof(struct perf_event_header);
if (size - sizeof(struct perf_event_header)) {
err = readn(self->fd, p, size - sizeof(struct perf_event_header));We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.
--- tools/perf/util/session.c | 34 ++++++++++++++++++++++++++-------- 1 files changed, 26 insertions(+), 8 deletions(-)
tools/perf/util/session.c +26 -8
@@ -1064,8 +1064,9 @@ volatile int session_done;static int __perf_session__process_pipe_events (struct perf_session *self,struct perf_tool *tool) { -union perf_event event; -uint32_t size; +union perf_event *event; +uint32_t size, cur_size =0 ; +void *buf =NULL ;int skip =0 ;u64 head;int err;
@@ -1074,8 +1075,14 @@ static int __perf_session__process_pipe_events(struct perf_session *self,perf_tool__fill_defaults (tool); head =0 ; + cur_size =sizeof (union perf_event ); + + buf =malloc (cur_size); +if (!buf) +return -errno; more: - err =readn (self->fd, &event,sizeof (struct perf_event_header )); + event = buf; + err =readn (self->fd, event,sizeof (struct perf_event_header ));if (err <=0 ) {if (err ==0 ) goto done;
@@ -1085,13 +1092,23 @@ static int __perf_session__process_pipe_events(struct perf_session *self, }if (self->header.needs_swap) -perf_event_header__bswap (&event.header); +perf_event_header__bswap (&event->header); - size = event.header.size; + size = event->header.size;if (size ==0 ) size =8 ; - p = &event; +if (size > cur_size) { +void *new =realloc (buf, size); +if (!new) { +pr_err ("failed to allocate memory to read event\n" ); + goto out_err; + } + buf = new; + cur_size = size; + event = buf; + } + p = event; p +=sizeof (struct perf_event_header );if (size -sizeof (struct perf_event_header )) {
@@ -1107,9 +1124,9 @@ static int __perf_session__process_pipe_events(struct perf_session *self, } } -if ((skip =perf_session__process_event (self, &event, tool, head)) <0 ) { +if ((skip =perf_session__process_event (self, event, tool, head)) <0 ) {pr_err ("%#" PRIx64" [%#x]: failed to process type: %d\n" , - head, event.header.size, event.header.type); + head, event->header.size, event->header.type); err = -EINVAL ; goto out_err; }
@@ -1124,6 +1141,7 @@ static int __perf_session__process_pipe_events(struct perf_session *self, done: err =0 ; out_err: +free (buf);perf_session__warn_about_errors (self, tool);perf_session_free_sample_buffers (self);return err;
Signature · 1 line
1.7.4.1
Re: [PATCH v2 2/5] perf tools: fix piped mode read code
Quoted 27 lines · Stephane Eranian
In
__perf_session__process_pipe_events(), there was a risk we would read more than what a unionperf_eventstruct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union
perf_event, yet it was used as the backing to the read in:union perf_event event; void *p; size = event->header.size; p =&event; p += sizeof(struct perf_event_header); if (size - sizeof(struct perf_event_header)) { err = readn(self->fd, p, size - sizeof(struct perf_event_header));We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.
Quoted 5 lines
Signed-off-by Stephane Eranian<[email protected]>--- tools/perf/util/session.c | 34 ++++++++++++++++++++++++++-------- 1 files changed, 26 insertions(+), 8 deletions(-)
I don't have a file/perf with different sized events, but the change makes sense.
[tip:perf/core] perf tools: Fix piped mode read code
In reply to Stephane EranianCommit-ID: 444d28663936e286c752f05feca44d6041b1fce4 Gitweb: http://git.kernel.org/tip/444d28663936e286c752f05feca44d6041b1fce4 Author: Stephane Eranian <[email protected]> AuthorDate: Tue, 15 May 2012 13:28:12 +0200 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Tue, 22 May 2012 12:59:52 -0300
perf tools: Fix piped mode read code
In __perf_session__process_pipe_events(), there was a risk we would read more than what a union perf_event struct can hold. this could happen in case, perf is reading a file which contains new record types it does not know about and which are larger than anything it knows about.
In general, perf is supposed to skip records it does not understand, but in pipe mode, those have to be read and ignored. The fixed size header contains the size of the record, but that size may be larger than union perf_event, yet it was used as the backing to the read in:
union perf_event event;
void *p;
size = event->header.size;
p = &event;
p += sizeof(struct perf_event_header);
if (size - sizeof(struct perf_event_header)) {
err = readn(self->fd, p, size - sizeof(struct perf_event_header));We fix this by allocating a buffer based on the size reported in the header. We reuse the buffer as much as we can. We realloc in case it becomes too small. In the common case, the performance impact is negligible.
--- tools/perf/util/session.c | 34 ++++++++++++++++++++++++++-------- 1 files changed, 26 insertions(+), 8 deletions(-)
tools/perf/util/session.c +26 -8
@@ -1098,8 +1098,9 @@ volatile int session_done;static int __perf_session__process_pipe_events (struct perf_session *self,struct perf_tool *tool) { -union perf_event event; -uint32_t size; +union perf_event *event; +uint32_t size, cur_size =0 ; +void *buf =NULL ;int skip =0 ;u64 head;int err;
@@ -1108,8 +1109,14 @@ static int __perf_session__process_pipe_events(struct perf_session *self,perf_tool__fill_defaults (tool); head =0 ; + cur_size =sizeof (union perf_event ); + + buf =malloc (cur_size); +if (!buf) +return -errno; more: - err =readn (self->fd, &event,sizeof (struct perf_event_header )); + event = buf; + err =readn (self->fd, event,sizeof (struct perf_event_header ));if (err <=0 ) {if (err ==0 ) goto done;
@@ -1119,13 +1126,23 @@ more: }if (self->header.needs_swap) -perf_event_header__bswap (&event.header); +perf_event_header__bswap (&event->header); - size = event.header.size; + size = event->header.size;if (size ==0 ) size =8 ; - p = &event; +if (size > cur_size) { +void *new =realloc (buf, size); +if (!new) { +pr_err ("failed to allocate memory to read event\n" ); + goto out_err; + } + buf = new; + cur_size = size; + event = buf; + } + p = event; p +=sizeof (struct perf_event_header );if (size -sizeof (struct perf_event_header )) {
@@ -1141,9 +1158,9 @@ more: } } -if ((skip =perf_session__process_event (self, &event, tool, head)) <0 ) { +if ((skip =perf_session__process_event (self, event, tool, head)) <0 ) {pr_err ("%#" PRIx64" [%#x]: failed to process type: %d\n" , - head, event.header.size, event.header.type); + head, event->header.size, event->header.type); err = -EINVAL ; goto out_err; }
@@ -1158,6 +1175,7 @@ more: done: err =0 ; out_err: +free (buf);perf_session__warn_about_errors (self, tool);perf_session_free_sample_buffers (self);return err;
PATCH v2 3/5 perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA
In reply to Stephane EranianTo match the PERF_RECORD_HEADER_TRACING_DATA record type. This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.
--- tools/perf/builtin-record.c | 4 ++-- tools/perf/util/header.c | 10 +++++----- tools/perf/util/header.h | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-)
tools/perf/builtin-record.c +2 -2
@@ -396,7 +396,7 @@ static void perf_record__mmap_read_all(struct perf_record *rec)perf_record__mmap_read (rec, &rec->evlist->mmap[i]); } -if (perf_header__has_feat (&rec->session->header,HEADER_TRACE_INFO )) +if (perf_header__has_feat (&rec->session->header,HEADER_TRACING_DATA ))write_output (rec, &finished_round_event,sizeof (finished_round_event)); }
@@ -478,7 +478,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)perf_header__clear_feat (&session->header,HEADER_BUILD_ID );if (!have_tracepoints (&evsel_list->entries)) -perf_header__clear_feat (&session->header,HEADER_TRACE_INFO ); +perf_header__clear_feat (&session->header,HEADER_TRACING_DATA );if (!rec->opts.branch_stack)perf_header__clear_feat (&session->header,HEADER_BRANCH_STACK );
tools/perf/util/header.c +5 -5
@@ -437,7 +437,7 @@ static bool perf_session__read_build_ids(struct perf_session *session, bool withreturn ret; } -static int write_trace_info (int fd,struct perf_header *h__used , +static int write_tracing_data (int fd,struct perf_header *h__used ,struct perf_evlist *evlist) {return read_tracing_data (fd, &evlist->entries);
@@ -1472,7 +1472,7 @@ static int perf_header__read_build_ids(struct perf_header *header,return err; } -static int process_trace_info(struct perf_file_section *section __unused, +static int process_tracing_data(struct perf_file_section *section __unused, struct perf_header *ph __unused, int feat __unused, int fd) {
@@ -1508,11 +1508,11 @@ struct feature_ops { .full_only = true }/* feature_ops not implemented: */ -#define print_trace_info NULL -#define print_build_id NULL +#define print_tracing_data NULL +#define print_build_id NULLstatic const struct feature_ops feat_ops[HEADER_LAST_FEATURE ] = { -FEAT_OPP (HEADER_TRACE_INFO , trace_info), +FEAT_OPP (HEADER_TRACING_DATA , tracing_data),FEAT_OPP (HEADER_BUILD_ID , build_id),FEAT_OPA (HEADER_HOSTNAME , hostname),FEAT_OPA (HEADER_OSRELEASE , osrelease),
tools/perf/util/header.h +1 -1
@@ -12,7 +12,7 @@enum {HEADER_RESERVED =0 ,/* always cleared */ HEADER_FIRST_FEATURE =1 , -HEADER_TRACE_INFO =1 , +HEADER_TRACING_DATA =1 ,HEADER_BUILD_ID ,HEADER_HOSTNAME ,
Signature · 1 line
1.7.4.1
Re: [PATCH v2 3/5] perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA
Quoted 6 lines · Stephane Eranian
To match the
PERF_RECORD_HEADER_TRACING_DATArecord type. This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.Signed-off-by Stephane Eranian<[email protected]>
[tip:perf/core] perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA
In reply to Stephane EranianCommit-ID: 2eeaaa095d155d47d47d5df07571105b8ae76ffd Gitweb: http://git.kernel.org/tip/2eeaaa095d155d47d47d5df07571105b8ae76ffd Author: Stephane Eranian <[email protected]> AuthorDate: Tue, 15 May 2012 13:28:13 +0200 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Tue, 22 May 2012 12:57:46 -0300
perf tools: rename HEADER_TRACE_INFO to HEADER_TRACING_DATA
To match the PERF_RECORD_HEADER_TRACING_DATA record type.
This is the same info as the one used for pipe mode whereas the other one is for regular file output. This will help in the later patch to add meta-data infos in pipe mode.
--- tools/perf/builtin-record.c | 4 ++-- tools/perf/util/header.c | 10 +++++----- tools/perf/util/header.h | 2 +- 3 files changed, 8 insertions(+), 8 deletions(-)
tools/perf/builtin-record.c +2 -2
@@ -396,7 +396,7 @@ static void perf_record__mmap_read_all(struct perf_record *rec)perf_record__mmap_read (rec, &rec->evlist->mmap[i]); } -if (perf_header__has_feat (&rec->session->header,HEADER_TRACE_INFO )) +if (perf_header__has_feat (&rec->session->header,HEADER_TRACING_DATA ))write_output (rec, &finished_round_event,sizeof (finished_round_event)); }
@@ -478,7 +478,7 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv)perf_header__clear_feat (&session->header,HEADER_BUILD_ID );if (!have_tracepoints (&evsel_list->entries)) -perf_header__clear_feat (&session->header,HEADER_TRACE_INFO ); +perf_header__clear_feat (&session->header,HEADER_TRACING_DATA );if (!rec->opts.branch_stack)perf_header__clear_feat (&session->header,HEADER_BRANCH_STACK );
tools/perf/util/header.c +5 -5
@@ -437,7 +437,7 @@ static bool perf_session__read_build_ids(struct perf_session *session, bool withreturn ret; } -static int write_trace_info (int fd,struct perf_header *h__used , +static int write_tracing_data (int fd,struct perf_header *h__used ,struct perf_evlist *evlist) {return read_tracing_data (fd, &evlist->entries);
@@ -1472,7 +1472,7 @@ out:return err; } -static int process_trace_info(struct perf_file_section *section __unused, +static int process_tracing_data(struct perf_file_section *section __unused, struct perf_header *ph __unused, int feat __unused, int fd) {
@@ -1508,11 +1508,11 @@ struct feature_ops { .full_only = true }/* feature_ops not implemented: */ -#define print_trace_info NULL -#define print_build_id NULL +#define print_tracing_data NULL +#define print_build_id NULLstatic const struct feature_ops feat_ops[HEADER_LAST_FEATURE ] = { -FEAT_OPP (HEADER_TRACE_INFO , trace_info), +FEAT_OPP (HEADER_TRACING_DATA , tracing_data),FEAT_OPP (HEADER_BUILD_ID , build_id),FEAT_OPA (HEADER_HOSTNAME , hostname),FEAT_OPA (HEADER_OSRELEASE , osrelease),
tools/perf/util/header.h +1 -1
@@ -12,7 +12,7 @@enum {HEADER_RESERVED =0 ,/* always cleared */ HEADER_FIRST_FEATURE =1 , -HEADER_TRACE_INFO =1 , +HEADER_TRACING_DATA =1 ,HEADER_BUILD_ID ,HEADER_HOSTNAME ,
PATCH v2 4/5 perf record: add meta-data support for pipe-mode
In reply to Stephane EranianThis patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -
Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.
The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for BUILD_ID and TRACING_DATA.
We have modified util/header.c such that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct called feat_fd which encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.
It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis. That is always the case with the example shown below. However, one could also do:
$ perf record -o - noploop 2 | perf inject -b >perf.data $ cat perf.data | perf report -i -
With this patch, it is possible to get:
$ perf record -o - noploop 2 | perf inject -b | perf report -i -
# ========
# captured on: Fri Jan 20 18:13:55 2012
# ========
#
# hostname : quad
# os release : 3.2.0-rc7-tip
# perf version : 3.2.0
# arch : x86_64
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
# cpuid : GenuineIntel,6,15,11
# total memory : 8092884 kB
...
# HEADER_CPU_TOPOLOGY info available, use -I to display
noploop for 2 seconds
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.084 MB - (~3677 samples) ]
99.80% noploop noploop [.] noploop
0.19% noploop [kernel.kallsyms] [k] radix_tree_gang_lookup--- tools/perf/builtin-annotate.c | 5 + tools/perf/builtin-inject.c | 1 + tools/perf/builtin-record.c | 7 + tools/perf/builtin-report.c | 7 +- tools/perf/util/event.c | 12 + tools/perf/util/event.h | 18 ++ tools/perf/util/header.c | 532 +++++++++++++++++++++++++++++------------ tools/perf/util/header.h | 9 + tools/perf/util/session.c | 17 ++- tools/perf/util/tool.h | 4 +- 10 files changed, 449 insertions(+), 163 deletions(-)
tools/perf/builtin-annotate.c +5 -0
@@ -247,6 +247,11 @@ int cmd_annotate(int argc, const char **argv, const char *prefix __used) .mmap = perf_event__process_mmap, .comm = perf_event__process_comm, .fork = perf_event__process_task, + .attr = perf_event__process_attr, + .event_type = perf_event__process_event_type, + .tracing_data = perf_event__process_tracing_data, + .build_id = perf_event__process_build_id, + .feature = perf_event__process_feature, .ordered_samples = true, .ordering_requires_timestamps = true, },
tools/perf/builtin-inject.c +1 -0
@@ -217,6 +217,7 @@ struct perf_tool perf_inject = { .event_type = perf_event__repipe_event_type_synth, .tracing_data = perf_event__repipe_tracing_data_synth, .build_id = perf_event__repipe_op2_synth, + .feature = perf_event__repipe_op2_synth, };extern volatile int session_done;
tools/perf/builtin-record.c +7 -0
@@ -531,6 +531,13 @@ static int __cmd_record(struct perf_record *rec, int argc, const char **argv) }if (opts->pipe_output) { + err =perf_event__synthesize_features (tool, session, evsel_list, + process_synthesized_event); +if (err <0 ) { +pr_err ("Couldn't synthesize features.\n" ); +return err; + } + err =perf_event__synthesize_attrs (tool, session, process_synthesized_event);if (err <0 ) {
tools/perf/builtin-report.c +4 -3
@@ -43,7 +43,6 @@ struct perf_report {bool force, use_tui, use_gtk, use_stdio;bool hide_unresolved;bool dont_use_callchains; -bool show_full_info;bool show_threads;bool inverted_callchain;struct perf_read_values show_threads_values;
@@ -358,7 +357,8 @@ static int __cmd_report(struct perf_report *rep) }if (use_browser <=0 ) -perf_session__fprintf_info (session, stdout, rep->show_full_info); +perf_session__fprintf_info (session, stdout, + rep->tool.show_all_features);if (rep->show_threads)perf_read_values_init (&rep->show_threads_values);
@@ -562,6 +562,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __used) .event_type = perf_event__process_event_type, .tracing_data = perf_event__process_tracing_data, .build_id = perf_event__process_build_id, + .feature = perf_event__process_feature, .ordered_samples = true, .ordering_requires_timestamps = true, },
@@ -625,7 +626,7 @@ int cmd_report(int argc, const char **argv, const char *prefix __used) "Look for files with symbols relative to this directory"),OPT_STRING ('C' ,"cpu" , &report.cpu_list,"cpu" ,"list of cpus to profile" ), -OPT_BOOLEAN ('I' ,"show-info" , &report.show_full_info, +OPT_BOOLEAN ('I' ,"show-info" , &report.tool.show_all_features,"Display extended information about perf.data file" ),OPT_BOOLEAN (0 ,"source" , &symbol_conf.annotate_src, "Interleave source code with assembly code (default)"),
tools/perf/util/event.c +12 -0
@@ -23,6 +23,18 @@ static const char *perf_event__names[] = { [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA", [PERF_RECORD_HEADER_BUILD_ID] = "BUILD_ID", [PERF_RECORD_FINISHED_ROUND] = "FINISHED_ROUND", + [PERF_RECORD_HEADER_HOSTNAME] = "HOSTNAME", + [PERF_RECORD_HEADER_OSRELEASE] = "OSRELEASE", + [PERF_RECORD_HEADER_VERSION] = "VERSION", + [PERF_RECORD_HEADER_ARCH] = "ARCH", + [PERF_RECORD_HEADER_NRCPUS] = "NRCPUS", + [PERF_RECORD_HEADER_CPUDESC] = "CPUDESC", + [PERF_RECORD_HEADER_CPUID] = "CPUID", + [PERF_RECORD_HEADER_TOTAL_MEM] = "TOTAL_MEM", + [PERF_RECORD_HEADER_CMDLINE] = "CMDLINE", + [PERF_RECORD_HEADER_EVENT_DESC] = "EVENT_DESC", + [PERF_RECORD_HEADER_CPU_TOPOLOGY] = "CPU_TOPOLOGY", + [PERF_RECORD_HEADER_NUMA_TOPOLOGY] = "NUMA_TOPOLOGY", };const char *perf_event__name (unsigned int id)
tools/perf/util/event.h +18 -0
@@ -100,6 +100,18 @@ enum perf_user_event_type { /* above any possible kernel type */PERF_RECORD_HEADER_TRACING_DATA =66 ,PERF_RECORD_HEADER_BUILD_ID =67 ,PERF_RECORD_FINISHED_ROUND =68 , +PERF_RECORD_HEADER_HOSTNAME =69 , +PERF_RECORD_HEADER_OSRELEASE =70 , +PERF_RECORD_HEADER_VERSION =71 , +PERF_RECORD_HEADER_ARCH =72 , +PERF_RECORD_HEADER_NRCPUS =73 , +PERF_RECORD_HEADER_CPUDESC =74 , +PERF_RECORD_HEADER_CPUID =75 , +PERF_RECORD_HEADER_TOTAL_MEM =76 , +PERF_RECORD_HEADER_CMDLINE =77 , +PERF_RECORD_HEADER_EVENT_DESC =78 , +PERF_RECORD_HEADER_CPU_TOPOLOGY =79 , +PERF_RECORD_HEADER_NUMA_TOPOLOGY =80 ,PERF_RECORD_HEADER_MAX };
@@ -126,6 +138,11 @@ struct tracing_data_event {u32 size; }; +struct feature_event { +struct perf_event_header header; +char data[];/* size bytes of raw data specific to the feature */ +}; +union perf_event {struct perf_event_header header;struct ip_event ip;
@@ -139,6 +156,7 @@ union perf_event {struct event_type_event event_type;struct tracing_data_event tracing_data;struct build_id_event build_id; +struct feature_event feat; };void perf_event__print_totals (void );
tools/perf/util/header.c +375 -157
1,076 lines not on this page · raw message
tools/perf/util/header.h +9 -0
@@ -98,6 +98,15 @@ int build_id_cache__add_s(const char *sbuild_id, const char *debugdir, const char *name, bool is_kallsyms);int build_id_cache__remove_s (const char *sbuild_id,const char *debugdir); +int perf_event__synthesize_features (struct perf_tool *tool, +struct perf_session *session, +struct perf_evlist *evlist, +perf_event__handler_t process); + +int perf_event__process_feature (struct perf_tool *tool, +union perf_event *event, +struct perf_session *session); +int perf_event__synthesize_attr (struct perf_tool *tool,struct perf_event_attr *attr,u16 ids,u64 *id,perf_event__handler_t process);
tools/perf/util/session.c +15 -2
@@ -390,6 +390,14 @@ static int process_event_type_stub(struct perf_tool *tool __used,return 0 ; } +static int process_feature_stub (struct perf_tool *tool__used , +union perf_event *event__used , +struct perf_session *session__used ) +{ +dump_printf (": unhandled!\n" ); +return 0 ; +} +static int process_finished_round (struct perf_tool *tool,union perf_event *event,struct perf_session *session);
@@ -428,6 +436,8 @@ static void perf_tool__fill_defaults(struct perf_tool *tool)else tool->finished_round = process_finished_round_stub; } +if (tool->feature ==NULL ) + tool->feature = process_feature_stub; }void mem_bswap_64 (void *src,int byte_size)
@@ -530,8 +540,8 @@ static perf_event__swap_op perf_event__swap_ops[] = { [PERF_RECORD_SAMPLE] = perf_event__all64_swap, [PERF_RECORD_HEADER_ATTR] = perf_event__hdr_attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = perf_event__event_type_swap, - [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap, [PERF_RECORD_HEADER_BUILD_ID] = NULL, + [PERF_RECORD_HEADER_TRACING_DATA] = perf_event__tracing_data_swap, [PERF_RECORD_HEADER_MAX] = NULL, };
@@ -947,6 +957,8 @@ static int perf_session__process_user_event(struct perf_session *session, unionreturn tool->build_id (tool, event, session);case PERF_RECORD_FINISHED_ROUND :return tool->finished_round (tool, event, session); + case PERF_RECORD_HEADER_HOSTNAME ... PERF_RECORD_HEADER_NUMA_TOPOLOGY: +return tool->feature (tool, event, session);default :return -EINVAL ; }
@@ -1509,6 +1521,7 @@ void perf_session__fprintf_info(struct perf_session *session, FILE *fp,fprintf (fp,"# ========\n" );fprintf (fp,"# captured on: %s" ,ctime (&st.st_ctime)); -perf_header__fprintf_info (session, fp, full);fprintf (fp,"# ========\n#\n" ); +if (!session->fd_pipe) +perf_header__fprintf_info (session, fp, full); }
tools/perf/util/tool.h +3 -1
@@ -42,9 +42,11 @@ struct perf_tool {event_synth_op tracing_data;event_simple_op event_type;event_op2 finished_round, - build_id; + build_id, + feature;bool ordered_samples;bool ordering_requires_timestamps; + bool show_all_features; };#endif /* __PERF_TOOL_H */
Signature · 1 line
1.7.4.1
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 52 lines · Stephane Eranian
This patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -
Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.
The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for
BUILD_IDandTRACING_DATA.We have modified
util/header.csuch that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct calledfeat_fdwhich encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis. That is always the case with the example shown below. However, one could also do:
$ perf record -o - noploop 2 | perf inject -b>perf.data $ cat perf.data | perf report -i -With this patch, it is possible to get:
$ perf record -o - noploop 2 | perf inject -b | perf report -i - # ======== # captured on: Fri Jan 20 18:13:55 2012 # ======== # # hostname : quad # os release : 3.2.0-rc7-tip # perf version : 3.2.0 # arch : x86_64 # nrcpus online : 4 # nrcpus avail : 4 # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz # cpuid : GenuineIntel,6,15,11 # total memory : 8092884 kB ... # HEADER_CPU_TOPOLOGY info available, use -I to display noploop for 2 seconds [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ] 99.80% noploop noploop [.] noploop 0.19% noploop [kernel.kallsyms] [k] radix_tree_gang_lookup
Quoted 1 line
Signed-off-by Stephane Eranian<[email protected]>
I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...
David
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 54 lines · David Ahern
Quoted 53 lines · Stephane Eranian
This patch adds the meta-data header support for perf record when used in pipe mode: perf record -o -
Up until now, meta-data was only available when perf record was used in "regular" mode, i.e., generating a perf.data file. For users depending on pipe mode, no host, event header information was gathered. This patch addresses this limitation.
The difficulty in pipe mode is that information needs to be written sequentially to the pipe. Meta data headers are usually generated (and also expected) at the beginning of the file (or piped output). To solve this problem, we introduce new synthetic record types, one for each meta-data type. The approach is similar to what is already used for
BUILD_IDandTRACING_DATA.We have modified
util/header.csuch that the same routines are used to generate and read the meta-data information regardless of pipe-mode vs. regular mode. To make this work, we added a new struct calledfeat_fdwhich encapsulates all the information necessary to read or write meta-data information to a file/pipe or from a file/pipe.It should be noted that there is a limitation with the current perf in terms of endianess in pipe mode. Perf assumes the records are generated using the same endianess during collection and analysis. That is always the case with the example shown below. However, one could also do:
$ perf record -o - noploop 2 | perf inject -b>perf.data $ cat perf.data | perf report -i -With this patch, it is possible to get:
$ perf record -o - noploop 2 | perf inject -b | perf report -i - # ======== # captured on: Fri Jan 20 18:13:55 2012 # ======== # # hostname : quad # os release : 3.2.0-rc7-tip # perf version : 3.2.0 # arch : x86_64 # nrcpus online : 4 # nrcpus avail : 4 # cpudesc : Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz # cpuid : GenuineIntel,6,15,11 # total memory : 8092884 kB ... # HEADER_CPU_TOPOLOGY info available, use -I to display noploop for 2 seconds [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.084 MB - (~3677 samples) ] 99.80% noploop noploop [.] noploop 0.19% noploop [kernel.kallsyms] [k] radix_tree_gang_lookup
Quoted 9 lines
Quoted 1 line
Signed-off-by Stephane Eranian<[email protected]>I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...
Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.
Quoted 1 line
David
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 7 lines · Stephane Eranian
Quoted 6 lines · David Ahern
I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...
Quoted 5 lines
Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.
Understood. I don't object to synthesized events.
Arnaldo: preferences?
David
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Em Fri, May 18, 2012 at 10:50:45AM -0600, David Ahern escreveu:
Quoted 14 lines · Stephane Eranian
Quoted 7 lines · David Ahern
Quoted 6 lines
I love the feature and it works nicely, but there has been push back on adding more synthesized events. Also the size of the patch is a bit much to take in. If there is no objection to the synthesized can you break the patch up -- e.g., introduce the events in one, synthesis and processing functions in another, plug into the commands, ...
Quoted 5 lines
Without new synthesized events, you cannot make this work. This is by construction
of the pipe mode. Meta-data has be be injected in the stream and therefore it needs
PERF_RECORD_* types.
Quoted 3 lines
Understood. I don't object to synthesized events.
Arnaldo: preferences?
PeterZ was the one objecting to adding more userspace only events, Peter?
- Arnaldo
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 1 line · Arnaldo Carvalho de Melo
PeterZ was the one objecting to adding more userspace only events,
Ah, yes I was ;-)
So uhm the argument was something like perf_event_type is a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.
Furthermore most (if not all) the userspace thingies were setup (like in this case single session meta-data) things. So they don't belong in the event stream at all.
Quoted 1 line
From what I can remember all this is somehow related to how data is
passed to scripts or so and since there's only a single stdin everything is stuffed over it.
For the data file we should simply create another section in the header; and I think that is how it works these days.
For the script muck, I've really no idea how all that works, but why can't you wrap the stuff in another layer; have a header up front that says the next N bytes are meta-data and after that there's the regular data stream.
Or take some inspiration from one of the many multi-stream stream formats out there like mpeg or ogg or whatever.
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 10 lines · Peter Zijlstra
Quoted 1 line · Arnaldo Carvalho de Melo
PeterZ was the one objecting to adding more userspace only events,
Ah, yes I was ;-)
So uhm the argument was something like
perf_event_typeis a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.
Yes, I did not start this but I understand why it was done that way.
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
So we need some namespace, so we can tag certain records as "synthetic" vs. kernel-generated. The way it was done is to assume that the kernel is not going to have more than 63 type of record types. That's where we run the risk of overlap one day. But for now we are at 10 kernel record types.
As such you need to inject certain header infos in the stream of kernel
Quoted 18 lines
Furthermore most (if not all) the userspace thingies were setup (like in this case single session meta-data) things. So they don't belong in the event stream at all.
From what I can remember all this is somehow related to how data is passed to scripts or so and since there's only a single stdin everything is stuffed over it.
For the data file we should simply create another section in the header; and I think that is how it works these days.
For the script muck, I've really no idea how all that works, but why can't you wrap the stuff in another layer; have a header up front that says the next N bytes are meta-data and after that there's the regular data stream.
Or take some inspiration from one of the many multi-stream stream formats out there like mpeg or ogg or whatever.
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Hi,
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 1 line · Arnaldo Carvalho de Melo
PeterZ was the one objecting to adding more userspace only events,
Ah, yes I was ;-)
So uhm the argument was something like
perf_event_typeis a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.
Quoted 9 lines
Yes, I did not start this but I understand why it was done that way.
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
IIUC, to add build-ids we should check all dso's whether hits or not, but it's not same as the case of meta-data since we can know it before processing. Am I missing something?
Thanks,
Namhyung
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Em Wed, May 23, 2012 at 09:45:12AM +0900, Namhyung Kim escreveu:
Quoted 22 lines · Stephane Eranian
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Arnaldo Carvalho de Melo
Quoted 2 lines
PeterZ was the one objecting to adding more userspace only events,
Ah, yes I was ;-)
So uhm the argument was something like
perf_event_typeis a kernel enum and userspace stealing space there is going to get us into trouble eventually since userspace doesn't register its types in our enum.Quoted 9 lines
Yes, I did not start this but I understand why it was done that way.
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
Quoted 3 lines
IIUC, to add build-ids we should check all dso's whether hits or not, but it's not same as the case of meta-data since we can know it before processing. Am I missing something?
To add build ids properly we should read it when the kernel loads a DSO, so that when PERF_RECORD_MMAP is sent to the event stream, the build id is there, so no need for post processing, etc.
The way we do it now is racy, long running sessions may experience DSO updating, etc.
I think we can figure out a way to avoid the seeks Stephane mentions so to get something done as Peter suggests.
But at the same time I don't think reusing the existing mechanism is such a big problem as Peter makes of it :-\
- Arnaldo
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 4 lines · Arnaldo Carvalho de Melo
But at the same time I don't think reusing the existing mechanism is such a big problem as Peter makes of it :-\
Probably not, but just know that I'll not take any such abuse into consideration when adding new record types.
That pipe mess is ill-designed and I don't care about it.
Peter Zijlstra
maintainerRe: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
In reply to Stephane EranianQuoted 3 lines · Stephane Eranian
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode.
I know that, I said as much. But it is quite possible to multiplex streams without the need for seeking, look at these MKV containers that contain video, multiple-audio and multiple-subtitle tracks.
The fact is, whomever wrote that pipe mode took a bonghit before touching the keyboard. They didn't even ask if we could reserve a range for userspace events, let alone consider if it was a good idea to begin with.
If we ever manage to get splice() working you can't keep that pipe mode working anyway.
Pushing everything over a single stream isn't a scalable solution anyway in the same way that writing everything into the one file isn't the best thing and would ideally be replaced as well.
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 5 lines · Peter Zijlstra
Quoted 3 lines · Stephane Eranian
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode.
Quoted 8 lines
I know that, I said as much. But it is quite possible to multiplex streams without the need for seeking, look at these MKV containers that contain video, multiple-audio and multiple-subtitle tracks.
The fact is, whomever wrote that pipe mode took a bonghit before touching the keyboard. They didn't even ask if we could reserve a range for userspace events, let alone consider if it was a good idea to begin with.
You're probably right, but the fact of the matter is, it's there now and we have lots of perf.data piling up all over the place. So you need to maintain compatibility with those. In my patch, I am just extending what's already there. I was not trying to rewrite the whole pipe mode + meta data logic (though I have done some of that already in the past).
Clearly pipe-mode was an after thought, but it is useful, I know people who use it. Obviously I use it, otherwise I would not necessarily have spent the time fixing the meta-data headers.
But I think, for the time being, my patch addresses an issue and does not prevent us for discussing a new file format for perf record, I have nothing against that.
Quoted 7 lines
If we ever manage to get
splice()working you can't keep that pipe mode working anyway.Pushing everything over a single stream isn't a scalable solution anyway in the same way that writing everything into the one file isn't the best thing and would ideally be replaced as well.
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
In reply to Stephane EranianQuoted 6 lines · Stephane Eranian
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?
struct pipe_data {
u32 length;
u32 type;
char data[0];
}It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.
David
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:
Quoted 18 lines · Stephane Eranian
Quoted 6 lines
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?
struct pipe_data { u32 length; u32 type; char data[0]; }
Quoted 4 lines
It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.
I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)
- Arnaldo
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Quoted 24 lines · Arnaldo Carvalho de Melo
Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:
Quoted 18 lines · Stephane Eranian
Quoted 6 lines
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?
struct pipe_data { u32 length; u32 type; char data[0]; }Quoted 4 lines
It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.
Quoted 5 lines
I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)
- Arnaldo
I don't know about that 'easy' comment; that patch is fairly long. Hence my question about whether synthesized events are acceptable before spending time on it. ;-)
David
Re: [PATCH v2 4/5] perf record: add meta-data support for pipe-mode
Em Thu, May 24, 2012 at 10:22:04AM -0600, David Ahern escreveu:
Quoted 25 lines · Arnaldo Carvalho de Melo
Quoted 24 lines
Em Thu, May 24, 2012 at 09:36:35AM -0600, David Ahern escreveu:
Quoted 22 lines · Stephane Eranian
Quoted 6 lines
The problem is that the headers as they are written to the file need seeking in the file to update the offset table. That is NOT possible when you operate in pipe mode. As such you need to inject the header infos very much like kernel PERF_RECORD_*. That is also why you have perf inject -b. Buildids are added at the end of the run in file mode, and that's another seek to the offset table if I recall correctly.
Perhaps I am being too simple minded here, but why not dump the features to the pipe as a series of structs when the perf session is created?
struct pipe_data { u32 length; u32 type; char data[0]; }It would require the features to be written to a buffer first to get the length, but that's manageable without too much code change. Endianness would need to be handled -- maybe a u8 flags at the beginning.
Quoted 5 lines
Quoted 3 lines
I guess this comes down to somebody prototyping it. The way its being done is just the easiest one, and one that at least Peter dislikes a lot ;-)
I don't know about that 'easy' comment; that patch is fairly long.
Easy in the sense that we didn't need to come up with any new mechanism to insert that meta data.
Quoted 2 lines
Hence my question about whether synthesized events are acceptable before spending time on it. ;-)
I think that we need to follow Peter suggestion, which looks to me similar to what you're suggesting, i.e. before the event stream, that will become multichannel anyway.
- Arnaldo
PATCH v2 5/5 perf: make perf buildid-list work better with pipe mode
In reply to Stephane EranianIn order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.
$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H noploop for 2 seconds [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.084 MB - (~3678 samples) ] 0000000000000000000000000000000000000000 [kernel.kallsyms] 3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploop
The reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream. The buildid for the kernel is provided by a BUILD_ID record. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately. In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.
So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.
--- tools/perf/builtin-buildid-list.c | 6 +++++- tools/perf/util/build-id.c | 2 ++ 2 files changed, 7 insertions(+), 1 deletions(-)
tools/perf/builtin-buildid-list.c +5 -1
@@ -84,7 +84,11 @@ static int perf_session__list_build_ids(void)if (filename__fprintf_build_id (session->filename, stdout)) goto out; -if (with_hits) +/* +* in pipe-mode, the only way to get the buildids is to parse +* the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID +*/ +if (with_hits || session->fd_pipe)perf_session__process_events (session, &build_id__mark_dso_hit_ops);perf_session__fprintf_dsos_buildid (session, stdout, with_hits);
tools/perf/util/build-id.c +2 -0
@@ -65,6 +65,8 @@ struct perf_tool build_id__mark_dso_hit_ops = { .mmap = perf_event__process_mmap, .fork = perf_event__process_task, .exit = perf_event__exit_del_thread, + .attr = perf_event__process_attr, + .build_id = perf_event__process_build_id, };char *dso__build_id_filename (struct dso *self,char *bf,size_t size)
Signature · 1 line
1.7.4.1
Re: [PATCH v2 5/5] perf: make perf buildid-list work better with pipe mode
Quoted 19 lines · Stephane Eranian
In order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.
$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H noploop for 2 seconds [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.084 MB - (~3678 samples) ] 0000000000000000000000000000000000000000 [kernel.kallsyms] 3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploopThe reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream. The buildid for the kernel is provided by a
BUILD_IDrecord. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately. In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.
Quoted 8 lines
So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.
Signed-off-by Stephane Eranian<[email protected]>--- tools/perf/builtin-buildid-list.c | 6 +++++- tools/perf/util/build-id.c | 2 ++ 2 files changed, 7 insertions(+), 1 deletions(-)
[tip:perf/core] perf buildid-list: Work better with pipe mode
In reply to Stephane EranianCommit-ID: 299c345208ae15fbc1e1c96dc1b6ac61e4e5da10 Gitweb: http://git.kernel.org/tip/299c345208ae15fbc1e1c96dc1b6ac61e4e5da10 Author: Stephane Eranian <[email protected]> AuthorDate: Tue, 15 May 2012 13:28:15 +0200 Committer: Arnaldo Carvalho de Melo <[email protected]> CommitDate: Tue, 22 May 2012 13:03:54 -0300
perf buildid-list: Work better with pipe mode
In order for perf buildid-list to work with pipe-mode files, it needs to process buildids and event attr structs.
$ perf record -o - noploop 2 | ./perf inject -b | perf buildid-list -i - -H noploop for 2 seconds [ perf record: Woken up 1 times to write data ] [ perf record: Captured and wrote 0.084 MB - (~3678 samples) ] 0000000000000000000000000000000000000000 [kernel.kallsyms] 3a0d0629efe74a8da3eeba372cdbd74ad9b8f5d5 /usr/local/bin/noploop
The reason [kernel.kallsyms] shows a 0 build-id comes from the way buildids are injected in the stream.
The buildid for the kernel is provided by a BUILD_ID record. The [kernel.kallsyms] is provided by a MMAP record. There is no clean and obvious way to link the two, unfortunately.
In regular mode, the kernel buildid is generated from reading the ELF image or kallsyms and perf knows to associate [kernel.kallsyms] to it. Later on, when perf processes the [kernel.kallsyms] MMAP record, it will already have a dso for it.
So for now, make sure perf buildid-list shows the buildids for everything but the kernel image.
--- tools/perf/builtin-buildid-list.c | 6 +++++- tools/perf/util/build-id.c | 2 ++ 2 files changed, 7 insertions(+), 1 deletions(-)
tools/perf/builtin-buildid-list.c +5 -1
@@ -84,7 +84,11 @@ static int perf_session__list_build_ids(void)if (filename__fprintf_build_id (session->filename, stdout)) goto out; -if (with_hits) +/* +* in pipe-mode, the only way to get the buildids is to parse +* the record stream. Buildids are stored as RECORD_HEADER_BUILD_ID +*/ +if (with_hits || session->fd_pipe)perf_session__process_events (session, &build_id__mark_dso_hit_ops);perf_session__fprintf_dsos_buildid (session, stdout, with_hits);
tools/perf/util/build-id.c +2 -0
@@ -65,6 +65,8 @@ struct perf_tool build_id__mark_dso_hit_ops = { .mmap = perf_event__process_mmap, .fork = perf_event__process_task, .exit = perf_event__exit_del_thread, + .attr = perf_event__process_attr, + .build_id = perf_event__process_build_id, };char *dso__build_id_filename (struct dso *self,char *bf,size_t size)
Hi, Stephane
Quoted 8 lines · Stephane Eranian
This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.
In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).
Thanks,
Namhyung
Quoted 12 lines · Namhyung Kim
Hi, Stephane
Quoted 8 lines · Stephane Eranian
This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.
Quoted 7 lines
In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).
Thanks,
Namhyung
It is there -- see perf-report header. It gets dropped on pipe mode b/c the meta-data is in the header.
David
Hi,
Quoted 17 lines · David Ahern
Quoted 16 lines · Namhyung Kim
Hi, Stephane
Quoted 8 lines · Stephane Eranian
This patch series adds meta-data support to perf record/report in pipe mode:
perf record -o - foo | perf inject -b | perf report -i -We had meta-data (information about host configuration, perf tool version,...) in regular (file) mode but it was lacking in pipe mode. This patch series fixes this.
In addition to this change, I think it'd be great if we keep this meta-data somewhere for later use - I guess perf report and annotate on TUI, at least, will need it too. Just a suggestion :).
Quoted 8 lines
Quoted 3 lines
Thanks,
NamhyungIt is there -- see perf-report header. It gets dropped on pipe mode b/c the meta-data is in the header.
David
Do you mean this?
struct perf_report {
struct perf_tool tool;
struct perf_session *session;
char const *input_name;
bool force, use_tui, use_gtk, use_stdio;
bool hide_unresolved;
bool dont_use_callchains;
bool show_full_info;
bool show_threads;
bool inverted_callchain;
struct perf_read_values show_threads_values;
const char *pretty_printing_style;
symbol_filter_t annotate_init;
const char *cpu_list;
const char *symbol_filter_str;
DECLARE_BITMAP(cpu_bitmap, MAX_NR_CPUS);
};Thanks,
Namhyung
See tools/perf/util/header.{ch}
Runtime example:
perf script -i /tmp/perf.data -I
# ========
# captured on: Tue May 15 20:22:15 2012 # ======== # # hostname : nxos-vdc-dev3 # os release : 3.4.0-rc7+ # perf version : 3.4.rc4.137.g978da3.dirty # arch : x86_64 # nrcpus online : 16 # nrcpus avail : 16 # cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz # cpuid : GenuineIntel,6,26,5 # total memory : 24680324 kB # cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop
...
'perf report --stdio' also shows it. It gets lost in the TUI.
David
Hi,
Quoted 20 lines · David Ahern
See
tools/perf/util/header.{ch}
Runtime example:perf script -i
/tmp/perf.data-I# ========
# captured on: Tue May 15 20:22:15 2012 # ======== # # hostname : nxos-vdc-dev3 # os release : 3.4.0-rc7+ # perf version : 3.4.rc4.137.g978da3.dirty # arch : x86_64 # nrcpus online : 16 # nrcpus avail : 16 # cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz # cpuid : GenuineIntel,6,26,5 # total memory : 24680324 kB # cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop
Quoted 4 lines
...
'perf report --stdio' also shows it. It gets lost in the TUI.
Yeah, I know. What I mean is keeping the info in memory after parsing a perf.data file so that it can be used for displaying it on a dialog in perf report TUI (on demand) or checking arch before annotation or ...
Thanks,
Namhyung
Em Tue, May 15, 2012 at 08:38:05PM -0600, David Ahern escreveu:
Quoted 20 lines
See
tools/perf/util/header.{ch}
Runtime example:perf script -i
/tmp/perf.data-I# ========
# captured on: Tue May 15 20:22:15 2012 # ======== # # hostname : nxos-vdc-dev3 # os release : 3.4.0-rc7+ # perf version : 3.4.rc4.137.g978da3.dirty # arch : x86_64 # nrcpus online : 16 # nrcpus avail : 16 # cpudesc : Intel(R) Xeon(R) CPU E5540 @ 2.53GHz # cpuid : GenuineIntel,6,26,5 # total memory : 24680324 kB # cmdline : /tmp/pbuild/perf record -fo /tmp/perf.data -- /tmp/noploop
Quoted 3 lines
...
'perf report --stdio' also shows it. It gets lost in the TUI.
Yeah, we need to add a splash screen with this info when the TUI starts and also ad a hotkey to show it at anytime.
- Arnaldo