perf: 'live mode'
cover perf: 'live mode'
Currently, a perf session entails two steps: first 'perf record' or 'perf trace record' records the perf data to disk, then 'perf report' or 'perf trace report' reads the saved data from disk and reports the results.
This experimental patchset makes some changes to perf that instead allow the perf data to be piped directly from the record step to the report step, without ever touching the disk.
This is especially useful for 'perf trace' - adding this capability means that the trace scripts are no longer relegated to simple post-processing, but can be run in a continuous 'live mode', forever processing the event stream and e.g. periodically dumping current results, essentially becoming special-purpose 'top' applications, or continuously scanning the event stream for arbitrarily complex conditions to flag, etc...
Being able to feed the event stream over a pipe also makes it possible to do things like trace over the network using e.g. netcat.
It turns out that perf is pretty close to being able to do this already, with the exception of the header data; most of the work of this patchset deals with changing that. It does so in a mainly additive way: it doesn't make any changes to the existing disk format or normal disk-mode processing, just adds special treatment for the cases when 'perf [trace] record' records to stdout or 'perf [trace] report reads from stdin.
Here are some quick examples. Basically using '-' as the filename to the -o and -i options send/read the data from stdout/stdin:
Here's a simple 'perf record' | 'perf report' run:
root@tropicana:~# perf record -o - -c 1 -f -a -M -R -e raw_syscalls:sys_enter | perf report -i -
^C# Samples: 120234
#
# Overhead Command Shared Object Symbol
# ........ ............... ......................... ......
#
98.65% perf libpthread-2.8.90.so [.] __read
0.46% perf libpthread-2.8.90.so [.] __write_nocancel
0.32% perf libpthread-2.8.90.so [.] __open_nocancel
0.29% perf libpthread-2.8.90.so [.] __libc_close
0.07% perf libc-2.8.90.so [.] 0x000000000a4b15
0.03% perf libc-2.8.90.so [.] __fxstat64
0.03% perf libc-2.8.90.so [.] __open
0.03% perf libc-2.8.90.so [.] close
0.02% Xorg libc-2.8.90.so [.] __select
0.01% Xorg libpthread-2.8.90.so [.] __read
0.01% perf libpthread-2.8.90.so [.] __libc_lseek
0.01% Xorg libc-2.8.90.so [.] setitimer
0.01% perf libc-2.8.90.so [.] __xstat64
0.01% firefox libpthread-2.8.90.so [.] __read
0.00% perf libc-2.8.90.so [.] llseek
0.00% Xorg libpthread-2.8.90.so [.] __restore_rt
0.00% Xorg libc-2.8.90.so [.] sigprocmask
0.00% Xorg libc-2.8.90.so [.] __poll
0.00% wterm libc-2.8.90.so [.] __read
0.00% gnome-screensav libpthread-2.8.90.so [.] __read
0.00% firefox libc-2.8.90.so [.] __poll
0.00% Xorg libc-2.8.90.so [.] writev
0.00% gnome-settings- libpthread-2.8.90.so [.] __read
0.00% wterm libc-2.8.90.so [.] __select
0.00% perf libc-2.8.90.so [.] statfs
0.00% mysqld libc-2.8.90.so [.] __select
0.00% firefox libpthread-2.8.90.so [.] __pthread_cond_timedwait
0.00% firefox libpthread-2.8.90.so [.] __lll_unlock_wake
0.00% firefox libpthread-2.8.90.so [.] 0x0000000000decb
0.00% firefox libc-2.8.90.so [.] __select
0.00% perf libc-2.8.90.so [.] getcwd
0.00% perf libc-2.8.90.so [.] mmap64
0.00% gnome-settings- libc-2.8.90.so [.] __poll
0.00% gnome-screensav libc-2.8.90.so [.] __poll
0.00% firefox libc-2.8.90.so [.] writev
0.00% apache2 libpthread-2.8.90.so [.] __waitpid
0.00% apache2 libc-2.8.90.so [.] __select
#
# (For a higher level overview, try: perf report --sort comm,dso)
#Included in this patchset are a couple of 'top' scripts, rwtop and sctop, that are essentially just modifications of existing scripts. Basically the original scripts were modified to add a 5 second timer. In the handler for the timer, the current output summary is printed, and the state is cleared and begun anew, ad infinitum.
Here are the new scripts as shown in the perf trace list:
root@tropicana:~# perf trace -l List of available trace scripts: workqueue-stats workqueue stats (ins/exe/create/destroy) wakeup-latency system-wide min/max/avg wakeup latency rw-by-file <comm> r/w activity for a program, by file rwtop system-wide r/w top failed-syscalls [comm] system-wide failed syscalls rw-by-pid system-wide r/w activity syscall-counts-by-pid [comm] system-wide syscall counts, by pid failed-syscalls-by-pid [comm] system-wide failed syscalls, by pid sctop [comm] syscall top syscall-counts [comm] system-wide syscall counts
And here's a few iterations of the output of the 'sctop' Python script:
root@tropicana:~# perf trace record sctop | perf trace report sctop perf trace started with Python script /root/libexec/perf-core/scripts/python/sctop.py
syscall events:
event count
---------------------------------------- -----------
2 179
3 158
0 79 78 24 5 13 1 3 16 2 298 1 137 1 79 1 72 1 8 1
syscall events:
event count
---------------------------------------- -----------
0 659287
202 1995
1 812 2 246 3 224 78 56 12 51 5 23 23 22 8 18 7 18 4 9 38 6 61 5 254 4 20 2 16 2 137 1 15 1 14 1 9 1
syscall events:
event count
---------------------------------------- -----------
0 647178
202 1209
1 324
12 52
7 29
14 6 16 4 23 3 3 3 72 2 21 1 11 1 9 1 5 1 4 1 2 1
.
.
.
Obviously, it would be better to have a real top-like display for these rather than a continuously scrolling mode like this, and of course it will be much more useful once we get the syscall name injection events going (the column on the left shows syscall numbers only).
Last, but not least, here's an example that uses netcat to run the 'rwtop' Perl script over the network:
On the system being traced:
root@tropicana:~# perf trace record rwtop | nc localhost 7777 ^C
On the system collecting the data and printing the output every 5 seconds:
root@tropicana:~# nc -l -p 7777 | perf trace report rwtop perf trace started with Perl script /root/libexec/perf-core/scripts/perl/rwtop.pl
read counts by pid:
pid comm # reads bytes_requested bytes_read
------ -------------------- ----------- ---------- ---------- 6672 perf 156016 6238975 6238980 6670 nc 741 6070272 6054460 6675 perf 3235 9445712 4589634 6674 nc 1376 11272192 1586892 6342 gvfsd-trash 2 4096 194 5856 Xorg 5 8264 76 6335 trashapplet 5 16384 48 6378 gnome-power-man 3 12288 32 6298 gnome-panel 1 4096 0
write counts by pid:
pid comm # writes bytes_written
------ -------------------- ----------- ---------- 6675 perf 15651 16412612 6670 nc 741 6054460 6674 nc 1376 1586892 6342 gvfsd-trash 2 93 6676 gvfsd-trash 1 35
read counts by pid:
pid comm # reads bytes_requested bytes_read
------ -------------------- ----------- ---------- ---------- 6672 perf 166458 6657104 6656232 6670 nc 787 6447104 6447104 6342 gvfsd-trash 2 4096 194 6335 trashapplet 5 16384 48
write counts by pid:
pid comm # writes bytes_written
------ -------------------- ----------- ---------- 6675 perf 13734 15206704 6670 nc 787 6447104 6342 gvfsd-trash 2 93 6677 gvfsd-trash 1 35
read counts by pid:
pid comm # reads bytes_requested bytes_read
------ -------------------- ----------- ---------- ---------- 6672 perf 167645 6704272 6704048 6670 nc 793 6496256 6529024 6298 gnome-panel 1 4096 0
write counts by pid:
pid comm # writes bytes_written
------ -------------------- ----------- ---------- 6675 perf 13473 15011968 6670 nc 797 6529024
failed to read event payload
read counts by pid:
pid comm # reads bytes_requested bytes_read
------ -------------------- ----------- ---------- ---------- 6670 nc 217 1777664 1875968 6672 perf 45782 1830816 1830560 6451 wterm 3 19873 607 5856 Xorg 3 72 72 6480 wterm 3 12288 32 6263 gnome-settings- 3 12288 32 6326 gnome-screensav 4 16384 32
write counts by pid:
pid comm # writes bytes_written
------ -------------------- ----------- ---------- 6675 perf 3821 4635048 6670 nc 229 1875968 6672 perf 11 592
perf trace Perl script stopped
There are some rough edges and inefficiencies, and I guess 'event injection' may be a better way to do this in the end, but it seems to work pretty well already, and I think it shows how useful such a capability can be.
Tom Zanussi (7):
perf: introduce special handling for pipe input/output
perf: add pipe-specific header read/write and event processing code
perf: convert perf header attrs into attr events
perf: convert perf event types into event type events
perf: convert perf tracing data into a tracing_data event
perf: convert perf header build_ids into build_id events perf trace/scripting: rwtop and sctop scripts
tools/perf/builtin-record.c | 66 +++++- tools/perf/builtin-report.c | 13 +- tools/perf/builtin-trace.c | 11 + tools/perf/scripts/perl/bin/rwtop-record | 2 + tools/perf/scripts/perl/bin/rwtop-report | 6 + tools/perf/scripts/perl/rwtop.pl | 165 +++++++++++++ tools/perf/scripts/python/bin/sctop-record | 2 + tools/perf/scripts/python/bin/sctop-report | 4 + tools/perf/scripts/python/sctop.py | 64 +++++ tools/perf/util/event.h | 35 +++ tools/perf/util/header.c | 348 +++++++++++++++++++++++++++- tools/perf/util/header.h | 38 +++- tools/perf/util/session.c | 214 ++++++++++++++++- tools/perf/util/session.h | 23 ++- tools/perf/util/trace-event-info.c | 24 ++ tools/perf/util/trace-event-read.c | 89 ++++---- tools/perf/util/trace-event.h | 4 +- 17 files changed, 1035 insertions(+), 73 deletions(-) create mode 100644 tools/perf/scripts/perl/bin/rwtop-record create mode 100644 tools/perf/scripts/perl/bin/rwtop-report create mode 100644 tools/perf/scripts/perl/rwtop.pl create mode 100644 tools/perf/scripts/python/bin/sctop-record create mode 100644 tools/perf/scripts/python/bin/sctop-report create mode 100644 tools/perf/scripts/python/sctop.py
RFC PATCH 1/7 perf: introduce special handling for pipe input/output
Adds special treatment for stdin/stdout - if the user specifies '-o -' to perf record or '-i -' to perf report et al, the intent is that the event stream be written to stdout or read from stdin rather than a disk file.
This initial patch just handles parsing the cmdline for those modes and setting a couple variables to be used by the code in later patches.
--- tools/perf/builtin-record.c | 10 ++++++++-- tools/perf/builtin-report.c | 3 ++- tools/perf/util/session.c | 10 ++++++++++ tools/perf/util/session.h | 1 + 4 files changed, 21 insertions(+), 3 deletions(-)
tools/perf/builtin-record.c +8 -2
@@ -35,6 +35,7 @@ static unsigned int page_size;static unsigned int mmap_pages =128 ;static int freq =1000 ;static int output; +static int pipe_output =0 ;static const char *output_name ="perf.data" ;static int group =0 ;static unsigned int realtime_prio =0 ;
@@ -431,7 +432,9 @@ static int __cmd_record(int argc, const char **argv)exit (-1 ); } -if (!stat (output_name, &st) && st.st_size) { +if (!strcmp (output_name,"-" )) + pipe_output =1 ; +else if (!stat (output_name, &st) && st.st_size) {if (!force) {if (!append_file) { pr_err("Error, output file %s exists, use -A "
@@ -456,7 +459,10 @@ static int __cmd_record(int argc, const char **argv)else flags |=O_TRUNC ; - output =open (output_name, flags,S_IRUSR |S_IWUSR ); +if (pipe_output) + output =STDOUT_FILENO ; +else + output =open (output_name, flags,S_IRUSR |S_IWUSR );if (output <0 ) {perror ("failed to create output file" );exit (-1 );
tools/perf/builtin-report.c +2 -1
@@ -359,7 +359,8 @@ int cmd_report(int argc, const char **argv, const char *prefix __used) { argc =parse_options (argc, argv, options, report_usage,0 ); -setup_pager (); +if (strcmp (input_name,"-" ) !=0 ) +setup_pager ();if (symbol__init () <0 )return -1 ;
tools/perf/util/session.c +10 -0
@@ -14,6 +14,16 @@ static int perf_session__open(struct perf_session *self, bool force) {struct stat input_stat; +if (!strcmp (self->filename,"-" )) { + self->fd_pipe = true; + self->fd =STDIN_FILENO ; + +if (perf_header__read (&self->header, self->fd) <0 ) +pr_err ("incompatible file format" ); + +return 0 ; + } + self->fd =open (self->filename,O_RDONLY );if (self->fd <0 ) {pr_err ("failed to open file: %s" , self->filename);
tools/perf/util/session.h +1 -0
@@ -26,6 +26,7 @@ struct perf_session {u64 sample_type;struct ref_reloc_sym ref_reloc_sym;int fd; +bool fd_pipe;int cwdlen;char *cwd;char filename[0 ];
Signature · 1 line
1.6.4.GIT
RFC PATCH 2/7 perf: add pipe-specific header read/write and event processing code
In reply to Tom ZanussiThis patch makes several changes to allow the perf event stream to be sent and received over a pipe:
- adds pipe-specific versions of the header read/write code
- adds pipe-specific version of the event processing code
- adds a range of event types to be used for header or other pseudo events, above the range used by the kernel
- adds code to interpret return values >0 from event processing functions as 'actions'. The first action defined is for stopping the event-processing loop at the first non-header event. So initially there's still a restriction that header events come first in the event stream.
Note that none of this affects the existing perf data file format or processing - this code only comes into play if perf output is sent to stdout (or is read from stdin).
--- tools/perf/builtin-record.c | 8 ++- tools/perf/builtin-report.c | 10 +++ tools/perf/builtin-trace.c | 11 +++ tools/perf/util/event.h | 4 + tools/perf/util/header.c | 62 ++++++++++++++++++- tools/perf/util/header.h | 8 ++- tools/perf/util/session.c | 145 +++++++++++++++++++++++++++++++++++++++--- tools/perf/util/session.h | 16 +++++ 8 files changed, 249 insertions(+), 15 deletions(-)
tools/perf/builtin-record.c +6 -2
@@ -475,7 +475,7 @@ static int __cmd_record(int argc, const char **argv) }if (!file_new) { - err =perf_header__read (&session->header, output); + err =perf_header__read (session, output);if (err <0 )return err; }
@@ -554,7 +554,11 @@ static int __cmd_record(int argc, const char **argv)open_counters (i, target_pid); } -if (file_new) { +if (pipe_output) { + err =perf_header__write_pipe (&session->header, output); +if (err <0 ) +return err; + }else if (file_new) { err =perf_header__write (&session->header, output, false);if (err <0 )return err;
tools/perf/builtin-report.c +10 -0
@@ -193,11 +193,21 @@ static struct perf_event_ops event_ops = { .read = process_read_event, }; +extern volatile int done; + +static void sig_handler (int sig __attribute__((__unused__))) +{ + done =1 ; +} +static int __cmd_report (void ) {int ret = -EINVAL ;struct perf_session *session; +signal (SIGCHLD , sig_handler); +signal (SIGINT , sig_handler); + session =perf_session__new (input_name,O_RDONLY , force);if (session ==NULL )return -ENOMEM ;
tools/perf/builtin-trace.c +11 -0
@@ -106,8 +106,18 @@ static struct perf_event_ops event_ops = { .comm = event__process_comm, }; +extern volatile int done; + +static void sig_handler(int sig __unused) +{ + done =1 ; +} +static int __cmd_trace (struct perf_session *session) { +signal (SIGCHLD , sig_handler); +signal (SIGINT , sig_handler); +return perf_session__process_events (session, &event_ops); }
@@ -609,6 +619,7 @@ int cmd_trace(int argc, const char **argv, const char *prefix __used)return -1 ; } +perf_header__read (session, input); err = scripting_ops->generate_script ("perf-trace" ); goto out; }
tools/perf/util/event.h +4 -0
@@ -83,6 +83,10 @@ struct build_id_event {char filename[]; }; +enum perf_header_event_type {/* above any possible kernel type */ +PERF_RECORD_HEADER_MAX =64 , +}; + typedef union event_union { struct perf_event_header header; struct ip_event ip;
tools/perf/util/header.c +61 -1
80 lines not on this page · raw message
tools/perf/util/header.h +7 -1
@@ -39,6 +39,11 @@ struct perf_file_header {DECLARE_BITMAP (adds_features,HEADER_FEAT_BITS ); }; +struct perf_file_header_pipe { +u64 magic; +u64 size; +}; +struct perf_header ; int perf_file_header__read(struct perf_file_header *self,
@@ -60,8 +65,9 @@ struct perf_header {int perf_header__init (struct perf_header *self);void perf_header__exit (struct perf_header *self); -int perf_header__read (struct perf_header *self,int fd); +int perf_header__read (struct perf_session *session,int fd);int perf_header__write (struct perf_header *self,int fd,bool at_exit); +int perf_header__write_pipe (struct perf_header *self,int fd);int perf_header__add_attr (struct perf_header *self,struct perf_header_attr *attr);
tools/perf/util/session.c +134 -11
215 lines not on this page · raw message
tools/perf/util/session.h +16 -0
@@ -46,6 +46,18 @@ struct perf_event_ops { unthrottle; }; +/* +* interpret process_event return values as actions. +* PERF_PROCESS_EVENT_MAX or greater is interpreted as 'skip ahead +* that many bytes'. Since that can never be less than +* sizeof(perf_event_header), 1 through sizeof(perf_event_header) can +* be interpreted as 'actions'. +*/ +enum perf_process_event_action { +PERF_PROCESS_EVENT_STOP =1 , +PERF_PROCESS_EVENT_MAX +}; +struct perf_session *perf_session__new (const char *filename,int mode,bool force);void perf_session__delete (struct perf_session *self);
@@ -86,4 +98,8 @@ static inline struct map * {return map_groups__new_module (&self->kmaps, start, filename); } + +void perf_session__update_sample_type (struct perf_session *self); +void perf_event_ops__fill_stop (struct perf_event_ops *handler); +#endif /* __PERF_SESSION_H */
Signature · 1 line
1.6.4.GIT
Re: [RFC PATCH 2/7] perf: add pipe-specific header read/write and event processing code
Quoted 10 lines · Tom Zanussi
tools/perf/util/header.c +9
-0+int perf_header__write_pipe (struct perf_header *self,int fd) +{ +struct perf_file_header_pipe f_header; +int err; + + f_header = (struct perf_file_header_pipe ){ + .magic =PERF_MAGIC , + .size =sizeof (f_header), + };
tools/perf/util/header.c out_free: · line 448
Quoted 10 lines
tools/perf/util/header.c +10
-0+ + err =do_write (fd, &f_header,sizeof (f_header)); +if (err <0 ) { +pr_debug ("failed to write perf pipe header\n" ); +return err; + } + + self->frozen =1 ; +return 0 ; +}
I actually wonder why you bother with a header in the pipe mode, since it only contains MAGIC and its size.
Re: [RFC PATCH 2/7] perf: add pipe-specific header read/write and event processing code
Em Sat, Mar 27, 2010 at 04:14:57AM +0100, Frederic Weisbecker escreveu:
Quoted 14 lines · Tom Zanussi
tools/perf/util/header.c out_free: · line 438
Quoted 10 lines
tools/perf/util/header.c +9
-0+int perf_header__write_pipe (struct perf_header *self,int fd) +{ +struct perf_file_header_pipe f_header; +int err; + + f_header = (struct perf_file_header_pipe ){ + .magic =PERF_MAGIC , + .size =sizeof (f_header), + };I actually wonder why you bother with a header in the pipe mode, since it only contains MAGIC and its size.
Because he wants to detect endianness :-)
- ARnaldo
Re: [RFC PATCH 2/7] perf: add pipe-specific header read/write and event processing code
Em Sat, Mar 27, 2010 at 07:57:47PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 16 lines
Em Sat, Mar 27, 2010 at 04:14:57AM +0100, Frederic Weisbecker escreveu:
Quoted 14 lines · Tom Zanussi
tools/perf/util/header.c out_free: · line 438
Quoted 10 lines
tools/perf/util/header.c +9
-0+int perf_header__write_pipe (struct perf_header *self,int fd) +{ +struct perf_file_header_pipe f_header; +int err; + + f_header = (struct perf_file_header_pipe ){ + .magic =PERF_MAGIC , + .size =sizeof (f_header), + };I actually wonder why you bother with a header in the pipe mode, since it only contains MAGIC and its size.
Quoted 1 line
Because he wants to detect endianness :-)
Elaborating: piping over netcat from a big endian to a little endian machine works because he used the ->needs_swap logic I implemented to allow cross platform analysis.
Its just a matter of having the DSOS with matching build-ids in the cache of the machine running 'perf report'.
- Arnaldo
Re: [RFC PATCH 2/7] perf: add pipe-specific header read/write and event processing code
Quoted 19 lines · Arnaldo Carvalho de Melo
Em Sat, Mar 27, 2010 at 07:57:47PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 16 lines
Em Sat, Mar 27, 2010 at 04:14:57AM +0100, Frederic Weisbecker escreveu:
Quoted 14 lines · Tom Zanussi
tools/perf/util/header.c out_free: · line 438
Quoted 10 lines
tools/perf/util/header.c +9
-0+int perf_header__write_pipe (struct perf_header *self,int fd) +{ +struct perf_file_header_pipe f_header; +int err; + + f_header = (struct perf_file_header_pipe ){ + .magic =PERF_MAGIC , + .size =sizeof (f_header), + };I actually wonder why you bother with a header in the pipe mode, since it only contains MAGIC and its size.
Quoted 1 line
Because he wants to detect endianness :-)
Quoted 8 lines
Elaborating: piping over netcat from a big endian to a little endian machine works because he used the ->needs_swap logic I implemented to allow cross platform analysis.
Its just a matter of having the DSOS with matching build-ids in the cache of the machine running 'perf report'.
- Arnaldo
Ok, thanks for the explanations :-)
Re: [RFC PATCH 2/7] perf: add pipe-specific header read/write and event processing code
Em Sun, Mar 28, 2010 at 12:12:05AM +0100, Frederic Weisbecker escreveu:
Quoted 8 lines · Arnaldo Carvalho de Melo
Quoted 7 lines
Em Sat, Mar 27, 2010 at 07:57:47PM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 5 lines
Em Sat, Mar 27, 2010 at 04:14:57AM +0100, Frederic Weisbecker escreveu:
Quoted 2 lines
I actually wonder why you bother with a header in the pipe mode, since it only contains MAGIC and its size.
Because he wants to detect endianness :-)
Quoted 8 lines
Quoted 6 lines
Elaborating: piping over netcat from a big endian to a little endian machine works because he used the ->needs_swap logic I implemented to allow cross platform analysis.
Its just a matter of having the DSOS with matching build-ids in the cache of the machine running 'perf report'.
Ok, thanks for the explanations :-)
Humm, now that I read this again, the build-ids now are in the headers, which will not be pushed as it is the last step in 'perf record', after we stop recording events, so only if we make sure that the binaries are the same is that we will be able to make sense of it over the network.
One more reason for the loader of DSOs to read the pre calculated build-id ELF session to tell the kernel the 20 bytes build id that in turn will inject it in the PERF_RECORD_MMAP event.
- Arnaldo
RFC PATCH 3/7 perf: convert perf header attrs into attr events
In reply to Tom ZanussiBypasses the attr perf header code and replaces it with a synthesized event and processing function that accomplishes the same thing, used when reading/writing perf data to/from a pipe.
Making the attrs into events allows them to be streamed over a pipe along with the rest of the header data (in later patches). It also paves the way to allowing events to be added and removed from perf sessions dynamically.
--- tools/perf/builtin-record.c | 10 +++++ tools/perf/util/event.h | 10 +++++- tools/perf/util/header.c | 80 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 8 ++++ tools/perf/util/session.c | 27 ++++++++++++++ tools/perf/util/session.h | 3 +- 6 files changed, 136 insertions(+), 2 deletions(-)
tools/perf/builtin-record.c +10 -0
@@ -566,6 +566,16 @@ static int __cmd_record(int argc, const char **argv) post_processing_offset =lseek (output,0 ,SEEK_CUR ); +if (pipe_output) { + err =event__synthesize_attrs (&session->header, + process_synthesized_event, + session); +if (err <0 ) { +pr_err ("Couldn't synthesize attrs.\n" ); +return err; + } + } + err =event__synthesize_kernel_mmap (process_synthesized_event, session,"_text" );if (err <0 ) {
tools/perf/util/event.h +9 -1
@@ -84,7 +84,14 @@ struct build_id_event { };enum perf_header_event_type {/* above any possible kernel type */ -PERF_RECORD_HEADER_MAX =64 , +PERF_RECORD_HEADER_ATTR =64 , +PERF_RECORD_HEADER_MAX +}; + +struct attr_event { +struct perf_event_header header; +struct perf_event_attr attr; +u64 id[]; }; typedef union event_union {
@@ -96,6 +103,7 @@ typedef union event_union {struct lost_event lost;struct read_event read;struct sample_event sample; +struct attr_event attr; } event_t;struct events_stats {
tools/perf/util/header.c +80 -0
89 lines not on this page · raw message
tools/perf/util/header.h +8 -0
@@ -95,4 +95,12 @@ int build_id_cache__add_s(const char *sbuild_id, const char *debugdir, const char *name, bool is_kallsyms);int build_id_cache__remove_s (const char *sbuild_id,const char *debugdir); +int event__synthesize_attr (struct perf_event_attr *attr,u16 ids,u64 *id, +event__handler_t process, +struct perf_session *session); +int event__synthesize_attrs (struct perf_header *self, +event__handler_t process, +struct perf_session *session); +int event__process_attr (event_t *self,struct perf_session *session); +#endif /* __PERF_HEADER_H */
tools/perf/util/session.c +27 -0
@@ -209,6 +209,8 @@ static void perf_event_ops__fill_defaults(struct perf_event_ops *handler) handler->throttle = process_event_stub;if (handler->unthrottle ==NULL ) handler->unthrottle = process_event_stub; +if (handler->attr ==NULL ) + handler->attr = process_event_stub; } static int process_event_stop(event_t *event __used,
@@ -229,6 +231,7 @@ void perf_event_ops__fill_stop(struct perf_event_ops *handler) handler->read = process_event_stop; handler->throttle = process_event_stop; handler->unthrottle = process_event_stop; + handler->attr = process_event_stop; }static const char *event__name[] = {
@@ -242,6 +245,7 @@ static const char *event__name[] = { [PERF_RECORD_FORK] = "FORK", [PERF_RECORD_READ] = "READ", [PERF_RECORD_SAMPLE] = "SAMPLE", + [PERF_RECORD_HEADER_ATTR] = "ATTR", };unsigned long event__total[PERF_RECORD_HEADER_MAX ];
@@ -308,6 +312,26 @@ static void event__read_swap(event_t *self) self->read.id =bswap_64 (self->read.id); } +static void event__attr_swap (event_t *self) +{ +size_t size; + + self->attr.attr.type =bswap_32 (self->attr.attr.type); + self->attr.attr.size =bswap_32 (self->attr.attr.size); + self->attr.attr.config =bswap_64 (self->attr.attr.config); + self->attr.attr.sample_period =bswap_64 (self->attr.attr.sample_period); + self->attr.attr.sample_type =bswap_64 (self->attr.attr.sample_type); + self->attr.attr.read_format =bswap_64 (self->attr.attr.read_format); + self->attr.attr.wakeup_events =bswap_32 (self->attr.attr.wakeup_events); + self->attr.attr.bp_type =bswap_32 (self->attr.attr.bp_type); + self->attr.attr.bp_addr =bswap_64 (self->attr.attr.bp_addr); + self->attr.attr.bp_len =bswap_64 (self->attr.attr.bp_len); + + size = self->header.size; + size -= (void *)&self->attr.id - (void *)self; +mem_bswap_64 (self->attr.id, size); +} +typedef void (*event__swap_op )(event_t *self);static event__swap_op event__swap_ops[] = {
@@ -318,6 +342,7 @@ static event__swap_op event__swap_ops[] = { [PERF_RECORD_LOST] = event__all64_swap, [PERF_RECORD_READ] = event__read_swap, [PERF_RECORD_SAMPLE] = event__all64_swap, + [PERF_RECORD_HEADER_ATTR] = event__attr_swap, [PERF_RECORD_HEADER_MAX] = NULL, };
@@ -358,6 +383,8 @@ static int perf_session__process_event(struct perf_session *self,return ops->throttle (event, self);case PERF_RECORD_UNTHROTTLE :return ops->unthrottle (event, self); +case PERF_RECORD_HEADER_ATTR : +return ops->attr (event, self);default : self->unknown_events++;return -1 ;
tools/perf/util/session.h +2 -1
@@ -43,7 +43,8 @@ struct perf_event_ops { lost, read, throttle, - unthrottle; + unthrottle, + attr; };/*
Signature · 1 line
1.6.4.GIT
RFC PATCH 4/7 perf: convert perf event types into event type events
In reply to Tom ZanussiBypasses the event type perf header code and replaces it with a synthesized event and processing function that accomplishes the same thing, used when reading/writing perf data to/from a pipe.
--- tools/perf/builtin-record.c | 7 +++++ tools/perf/util/event.h | 14 +++++++++ tools/perf/util/header.c | 63 ++++++++++++++++++++++++++++++++++++++----- tools/perf/util/header.h | 9 ++++++ tools/perf/util/session.c | 13 +++++++++ tools/perf/util/session.h | 3 +- 6 files changed, 101 insertions(+), 8 deletions(-)
tools/perf/builtin-record.c +7 -0
@@ -574,6 +574,13 @@ static int __cmd_record(int argc, const char **argv)pr_err ("Couldn't synthesize attrs.\n" );return err; } + + err =event__synthesize_event_types (process_synthesized_event, + session); +if (err <0 ) { +pr_err ("Couldn't synthesize event_types.\n" ); +return err; + } } err = event__synthesize_kernel_mmap(process_synthesized_event,
tools/perf/util/event.h +14 -0
@@ -85,6 +85,7 @@ struct build_id_event {enum perf_header_event_type {/* above any possible kernel type */ PERF_RECORD_HEADER_ATTR =64 , +PERF_RECORD_HEADER_EVENT_TYPE =65 ,PERF_RECORD_HEADER_MAX };
@@ -94,6 +95,18 @@ struct attr_event {u64 id[]; }; +#define MAX_EVENT_NAME 64 + +struct perf_trace_event_type { +u64 event_id; +char name[MAX_EVENT_NAME ]; +}; + +struct event_type_event { +struct perf_event_header header; +struct perf_trace_event_type event_type; +}; + typedef union event_union { struct perf_event_header header; struct ip_event ip;
@@ -104,6 +117,7 @@ typedef union event_union {struct read_event read;struct sample_event sample;struct attr_event attr; +struct event_type_event event_type; } event_t;struct events_stats {
tools/perf/util/header.c +56 -7
78 lines not on this page · raw message
tools/perf/util/header.h +9 -0
@@ -103,4 +103,13 @@ int event__synthesize_attrs(struct perf_header *self, struct perf_session *session);int event__process_attr (event_t *self,struct perf_session *session); +int event__synthesize_event_type (u64 event_id,char *name, +event__handler_t process, +struct perf_session *session); +int event__synthesize_event_types (event__handler_t process, +struct perf_session *session); +int event__process_event_type (event_t *self, +struct perf_session *session); + +#endif /* __PERF_HEADER_H */
tools/perf/util/session.c +13 -0
@@ -211,6 +211,8 @@ static void perf_event_ops__fill_defaults(struct perf_event_ops *handler) handler->unthrottle = process_event_stub;if (handler->attr ==NULL ) handler->attr = process_event_stub; +if (handler->event_type ==NULL ) + handler->event_type = process_event_stub; } static int process_event_stop(event_t *event __used,
@@ -232,6 +234,7 @@ void perf_event_ops__fill_stop(struct perf_event_ops *handler) handler->throttle = process_event_stop; handler->unthrottle = process_event_stop; handler->attr = process_event_stop; + handler->event_type = process_event_stop; }static const char *event__name[] = {
@@ -246,6 +249,7 @@ static const char *event__name[] = { [PERF_RECORD_READ] = "READ", [PERF_RECORD_SAMPLE] = "SAMPLE", [PERF_RECORD_HEADER_ATTR] = "ATTR", + [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", };unsigned long event__total[PERF_RECORD_HEADER_MAX ];
@@ -332,6 +336,12 @@ static void event__attr_swap(event_t *self)mem_bswap_64 (self->attr.id, size); } +static void event__event_type_swap (event_t *self) +{ + self->event_type.event_type.event_id = +bswap_64 (self->event_type.event_type.event_id); +} +typedef void (*event__swap_op )(event_t *self);static event__swap_op event__swap_ops[] = {
@@ -343,6 +353,7 @@ static event__swap_op event__swap_ops[] = { [PERF_RECORD_READ] = event__read_swap, [PERF_RECORD_SAMPLE] = event__all64_swap, [PERF_RECORD_HEADER_ATTR] = event__attr_swap, + [PERF_RECORD_HEADER_EVENT_TYPE] = event__event_type_swap, [PERF_RECORD_HEADER_MAX] = NULL, };
@@ -385,6 +396,8 @@ static int perf_session__process_event(struct perf_session *self,return ops->unthrottle (event, self);case PERF_RECORD_HEADER_ATTR :return ops->attr (event, self); +case PERF_RECORD_HEADER_EVENT_TYPE : +return ops->event_type (event, self);default : self->unknown_events++;return -1 ;
tools/perf/util/session.h +2 -1
@@ -44,7 +44,8 @@ struct perf_event_ops { read, throttle, unthrottle, - attr; + attr, + event_type; };/*
Signature · 1 line
1.6.4.GIT
RFC PATCH 5/7 perf: convert perf tracing data into a tracing_data event
In reply to Tom ZanussiBypasses the tracing_data perf header code and replaces it with a synthesized event and processing function that accomplishes the same thing, used when reading/writing perf data to/from a pipe.
The tracing data is pretty large, and this patch doesn't attempt to break it down into component events. The tracing_data event itself doesn't actually contain the tracing data, rather it arranges for the event processing code to skip over it after it's read, using the 'action' added in a previous patch; in this case the action is 'skip'.
--- tools/perf/builtin-record.c | 16 ++++++ tools/perf/util/event.h | 7 +++ tools/perf/util/header.c | 53 +++++++++++++++++++++ tools/perf/util/header.h | 6 ++ tools/perf/util/session.c | 14 ++++++ tools/perf/util/session.h | 3 +- tools/perf/util/trace-event-info.c | 24 ++++++++++ tools/perf/util/trace-event-read.c | 89 ++++++++++++++++++------------------ tools/perf/util/trace-event.h | 4 +- 9 files changed, 170 insertions(+), 46 deletions(-)
tools/perf/builtin-record.c +16 -0
@@ -101,6 +101,11 @@ static void mmap_write_tail(struct mmap_data *md, unsigned long tail) pc->data_tail = tail; } +static void advance_output (size_t size) +{ + bytes_written += size; +} +static void write_output (void *buf,size_t size) {while (size) {
@@ -581,6 +586,17 @@ static int __cmd_record(int argc, const char **argv)pr_err ("Couldn't synthesize event_types.\n" );return err; } + + err =event__synthesize_tracing_data (output, attrs, + nr_counters, + process_synthesized_event, + session); +if (err <=0 ) { +pr_err ("Couldn't record tracing data.\n" ); +return err; + } + +advance_output (err); } err = event__synthesize_kernel_mmap(process_synthesized_event,
tools/perf/util/event.h +7 -0
@@ -86,6 +86,7 @@ struct build_id_event {enum perf_header_event_type {/* above any possible kernel type */ PERF_RECORD_HEADER_ATTR =64 ,PERF_RECORD_HEADER_EVENT_TYPE =65 , +PERF_RECORD_HEADER_TRACING_DATA =66 ,PERF_RECORD_HEADER_MAX };
@@ -107,6 +108,11 @@ struct event_type_event {struct perf_trace_event_type event_type; }; +struct tracing_data_event { +struct perf_event_header header; +u32 size; +}; + typedef union event_union { struct perf_event_header header; struct ip_event ip;
@@ -118,6 +124,7 @@ typedef union event_union {struct sample_event sample;struct attr_event attr;struct event_type_event event_type; +struct tracing_data_event tracing_data; } event_t;struct events_stats {
tools/perf/util/header.c +53 -0
62 lines not on this page · raw message
tools/perf/util/header.h +6 -0
@@ -111,5 +111,11 @@ int event__synthesize_event_types(event__handler_t process,int event__process_event_type (event_t *self,struct perf_session *session); +int event__synthesize_tracing_data (int fd,struct perf_event_attr *pattrs, +int nb_events, +event__handler_t process, +struct perf_session *session); +int event__process_tracing_data (event_t *self, +struct perf_session *session);#endif /* __PERF_HEADER_H */
tools/perf/util/session.c +14 -0
@@ -213,6 +213,8 @@ static void perf_event_ops__fill_defaults(struct perf_event_ops *handler) handler->attr = process_event_stub;if (handler->event_type ==NULL ) handler->event_type = process_event_stub; +if (handler->tracing_data ==NULL ) + handler->tracing_data = process_event_stub; } static int process_event_stop(event_t *event __used,
@@ -235,6 +237,7 @@ void perf_event_ops__fill_stop(struct perf_event_ops *handler) handler->unthrottle = process_event_stop; handler->attr = process_event_stop; handler->event_type = process_event_stop; + handler->tracing_data = process_event_stop; }static const char *event__name[] = {
@@ -250,6 +253,7 @@ static const char *event__name[] = { [PERF_RECORD_SAMPLE] = "SAMPLE", [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", + [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA", };unsigned long event__total[PERF_RECORD_HEADER_MAX ];
@@ -342,6 +346,11 @@ static void event__event_type_swap(event_t *self)bswap_64 (self->event_type.event_type.event_id); } +static void event__tracing_data_swap (event_t *self) +{ + self->tracing_data.size =bswap_32 (self->tracing_data.size); +} +typedef void (*event__swap_op )(event_t *self);static event__swap_op event__swap_ops[] = {
@@ -354,6 +363,7 @@ static event__swap_op event__swap_ops[] = { [PERF_RECORD_SAMPLE] = event__all64_swap, [PERF_RECORD_HEADER_ATTR] = event__attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = event__event_type_swap, + [PERF_RECORD_HEADER_TRACING_DATA] = event__tracing_data_swap, [PERF_RECORD_HEADER_MAX] = NULL, };
@@ -398,6 +408,10 @@ static int perf_session__process_event(struct perf_session *self,return ops->attr (event, self);case PERF_RECORD_HEADER_EVENT_TYPE :return ops->event_type (event, self); +case PERF_RECORD_HEADER_TRACING_DATA : +/* setup for reading amidst mmap */ +lseek (self->fd, offset + head,SEEK_SET ); +return ops->tracing_data (event, self);default : self->unknown_events++;return -1 ;
tools/perf/util/session.h +2 -1
@@ -45,7 +45,8 @@ struct perf_event_ops { throttle, unthrottle, attr, - event_type; + event_type, + tracing_data; };/*
tools/perf/util/trace-event-info.c +24 -0
@@ -154,10 +154,17 @@ static void put_tracing_file(char *file)free (file); } +static ssize_t calc_data_size; +static ssize_t write_or_die (const void *buf,size_t len) {int ret; +if (calc_data_size) { + calc_data_size += len; +return len; + } + ret =write (output_fd, buf, len);if (ret <0 )die ("writing to '%s'" , output_file);
@@ -526,3 +533,20 @@ int read_tracing_data(int fd, struct perf_event_attr *pattrs, int nb_events)return 0 ; } + +ssize_t read_tracing_data_size (int fd,struct perf_event_attr *pattrs, +int nb_events) +{ +ssize_t size; +int err =0 ; + + calc_data_size =1 ; + err =read_tracing_data (fd, pattrs, nb_events); + size = calc_data_size -1 ; + calc_data_size =0 ; + +if (err <0 ) +return err; + +return size; +}
tools/perf/util/trace-event-read.c +45 -44
142 lines not on this page · raw message
tools/perf/util/trace-event.h +3 -1
@@ -162,7 +162,7 @@ struct record *trace_read_data(int cpu);void parse_set_info (int nr_cpus,int long_sz); -void trace_report (int fd); +ssize_t trace_report (int fd);void *malloc_or_die (unsigned int size);
@@ -258,6 +258,8 @@ void *raw_field_ptr(struct event *event, const char *name, void *data);unsigned long long eval_flag (const char *flag);int read_tracing_data (int fd,struct perf_event_attr *pattrs,int nb_events); +ssize_t read_tracing_data_size (int fd,struct perf_event_attr *pattrs, +int nb_events);/* taken from kernel/trace/trace.h */ enum trace_flag_type {
Signature · 1 line
1.6.4.GIT
RFC PATCH 6/7 perf: convert perf header build_ids into build_id events
In reply to Tom ZanussiBypasses the build_id perf header code and replaces it with a synthesized event and processing function that accomplishes the same thing, used when reading/writing perf data to/from a pipe.
--- tools/perf/builtin-record.c | 15 ++++++- tools/perf/util/event.h | 2 + tools/perf/util/header.c | 90 +++++++++++++++++++++++++++++++++++++++++++ tools/perf/util/header.h | 7 +++ tools/perf/util/session.c | 7 +++ tools/perf/util/session.h | 3 +- 6 files changed, 120 insertions(+), 4 deletions(-)
tools/perf/builtin-record.c +12 -3
@@ -405,10 +405,19 @@ static int process_buildids(void)static void atexit_header (void ) { - session->header.data_size += bytes_written; +if (!pipe_output) { + session->header.data_size += bytes_written; -process_buildids (); -perf_header__write (&session->header, output, true); +process_buildids (); +perf_header__write (&session->header, output, true); + }else { +int err; + + err =event__synthesize_build_ids (process_synthesized_event, + session); +if (err <0 ) +pr_err ("Couldn't synthesize build ids.\n" ); + } }static int __cmd_record (int argc,const char **argv)
tools/perf/util/event.h +2 -0
@@ -87,6 +87,7 @@ enum perf_header_event_type { /* above any possible kernel type */PERF_RECORD_HEADER_ATTR =64 ,PERF_RECORD_HEADER_EVENT_TYPE =65 ,PERF_RECORD_HEADER_TRACING_DATA =66 , +PERF_RECORD_HEADER_BUILD_ID =67 , PERF_RECORD_HEADER_MAX };
@@ -125,6 +126,7 @@ typedef union event_union {struct attr_event attr;struct event_type_event event_type;struct tracing_data_event tracing_data; +struct build_id_event build_id; } event_t;struct events_stats {
tools/perf/util/header.c +90 -0
93 lines not on this page · raw message
tools/perf/util/header.h +7 -0
@@ -118,4 +118,11 @@ int event__synthesize_tracing_data(int fd, struct perf_event_attr *pattrs,int event__process_tracing_data (event_t *self,struct perf_session *session); +int event__synthesize_build_id (struct dso *pos,u16 misc, +event__handler_t process, +struct perf_session *session); +int event__synthesize_build_ids (event__handler_t process, +struct perf_session *session); +int event__process_build_id (event_t *self,struct perf_session *session); +#endif /* __PERF_HEADER_H */
tools/perf/util/session.c +7 -0
@@ -215,6 +215,8 @@ static void perf_event_ops__fill_defaults(struct perf_event_ops *handler) handler->event_type = process_event_stub;if (handler->tracing_data ==NULL ) handler->tracing_data = process_event_stub; +if (handler->build_id ==NULL ) + handler->build_id = process_event_stub; } static int process_event_stop(event_t *event __used,
@@ -238,6 +240,7 @@ void perf_event_ops__fill_stop(struct perf_event_ops *handler) handler->attr = process_event_stop; handler->event_type = process_event_stop; handler->tracing_data = process_event_stop; + handler->build_id = process_event_stop; }static const char *event__name[] = {
@@ -254,6 +257,7 @@ static const char *event__name[] = { [PERF_RECORD_HEADER_ATTR] = "ATTR", [PERF_RECORD_HEADER_EVENT_TYPE] = "EVENT_TYPE", [PERF_RECORD_HEADER_TRACING_DATA] = "TRACING_DATA", + [PERF_RECORD_HEADER_BUILD_ID] = "BUILD_ID", };unsigned long event__total[PERF_RECORD_HEADER_MAX ];
@@ -364,6 +368,7 @@ static event__swap_op event__swap_ops[] = { [PERF_RECORD_HEADER_ATTR] = event__attr_swap, [PERF_RECORD_HEADER_EVENT_TYPE] = event__event_type_swap, [PERF_RECORD_HEADER_TRACING_DATA] = event__tracing_data_swap, + [PERF_RECORD_HEADER_BUILD_ID] = NULL, [PERF_RECORD_HEADER_MAX] = NULL, };
@@ -412,6 +417,8 @@ static int perf_session__process_event(struct perf_session *self,/* setup for reading amidst mmap */ lseek (self->fd, offset + head,SEEK_SET );return ops->tracing_data (event, self); +case PERF_RECORD_HEADER_BUILD_ID : +return ops->build_id (event, self);default : self->unknown_events++;return -1 ;
tools/perf/util/session.h +2 -1
@@ -46,7 +46,8 @@ struct perf_event_ops { unthrottle, attr, event_type, - tracing_data; + tracing_data, + build_id; };/*
Signature · 1 line
1.6.4.GIT
RFC PATCH 7/7 perf trace/scripting: rwtop and sctop scripts
In reply to Tom ZanussiA couple of scripts, one in Python and the other in Perl, that demonstrate 'live mode' tracing. For each, the output of the perf event stream is fed continuously to the script, which continuously aggregates the data and reports the current results every 5 seconds. After the current results are displayed, the aggregations are cleared and the cycle begins anew.
To run the scripts, simply pipe the output of the 'perf trace record' step as input to the corresponding 'perf trace report' step:
$ perf trace record sctop | perf trace report sctop
A future patch should probably automatically do this i.e. the user should be able to just run 'perf trace sctop' and get the same thing, and the output should do better than just continuously scroll, but these are just demo scripts for now...
--- tools/perf/scripts/perl/bin/rwtop-record | 2 + tools/perf/scripts/perl/bin/rwtop-report | 6 + tools/perf/scripts/perl/rwtop.pl | 165 ++++++++++++++++++++++++++++ tools/perf/scripts/python/bin/sctop-record | 2 + tools/perf/scripts/python/bin/sctop-report | 4 + tools/perf/scripts/python/sctop.py | 64 +++++++++++ 6 files changed, 243 insertions(+), 0 deletions(-) create mode 100644 tools/perf/scripts/perl/bin/rwtop-record create mode 100644 tools/perf/scripts/perl/bin/rwtop-report create mode 100644 tools/perf/scripts/perl/rwtop.pl create mode 100644 tools/perf/scripts/python/bin/sctop-record create mode 100644 tools/perf/scripts/python/bin/sctop-report create mode 100644 tools/perf/scripts/python/sctop.py
tools/perf/scripts/perl/bin/rwtop-record new file +2 -0
@@ -0,0 +1,2 @@ +#!/bin/bash +perf record -o - -c 1 -f -a -M -R -e syscalls:sys_enter_read -e syscalls:sys_exit_read -e syscalls:sys_enter_write -e syscalls:sys_exit_write
tools/perf/scripts/perl/bin/rwtop-report new file +6 -0
@@ -0,0 +1,6 @@ +#!/bin/bash +# description: system-wide r/w top +perf trace -i - -s ~/libexec/perf-core/scripts/perl/rwtop.pl + + +
tools/perf/scripts/perl/rwtop.pl new file +165 -0
165 lines not on this page · raw message
tools/perf/scripts/python/bin/sctop-record new file +2 -0
@@ -0,0 +1,2 @@ +#!/bin/bash +perf record -o - -c 1 -f -a -M -R -e raw_syscalls:sys_enter
tools/perf/scripts/python/bin/sctop-report new file +4 -0
@@ -0,0 +1,4 @@ +#!/bin/bash +# description: syscall top +# args: [comm] +perf trace -i - -s ~/libexec/perf-core/scripts/python/sctop.py $1
tools/perf/scripts/python/sctop.py new file +64 -0
64 lines not on this page · raw message
Signature · 1 line
1.6.4.GIT
Quoted 7 lines · Tom Zanussi
Currently, a perf session entails two steps: first 'perf record' or 'perf trace record' records the perf data to disk, then 'perf report' or 'perf trace report' reads the saved data from disk and reports the results.
This experimental patchset makes some changes to perf that instead allow the perf data to be piped directly from the record step to the report step, without ever touching the disk.
Very nice!
Quoted 4 lines
Obviously, it would be better to have a real top-like display for these rather than a continuously scrolling mode like this, and of course it will be much more useful once we get the syscall name injection events going (the column on the left shows syscall numbers only).
It's still useful for ad-hoc tracing!
Side-note, it might make sense to expose the 'clear' escape sequence somehow in an easy fashion:
[H[2J
to make it non-scrolling ;-) via a pre-provided method.
Quoted 3 lines
There are some rough edges and inefficiencies, and I guess 'event injection' may be a better way to do this in the end, but it seems to work pretty well already, and I think it shows how useful such a capability can be.
I'm all for it. Frederic, Steve, what do you think?
Ingo
Em Thu, Mar 04, 2010 at 12:18:56PM +0100, Ingo Molnar escreveu:
Quoted 24 lines · Tom Zanussi
Quoted 8 lines
Currently, a perf session entails two steps: first 'perf record' or 'perf trace record' records the perf data to disk, then 'perf report' or 'perf trace report' reads the saved data from disk and reports the results.
This experimental patchset makes some changes to perf that instead allow the perf data to be piped directly from the record step to the report step, without ever touching the disk.
Very nice!
Quoted 4 lines
Obviously, it would be better to have a real top-like display for these rather than a continuously scrolling mode like this, and of course it will be much more useful once we get the syscall name injection events going (the column on the left shows syscall numbers only).
It's still useful for ad-hoc tracing!
Side-note, it might make sense to expose the 'clear' escape sequence somehow in an easy fashion:
[H[2J
Quoted 7 lines
to make it non-scrolling ;-) via a pre-provided method.
Quoted 3 lines
There are some rough edges and inefficiencies, and I guess 'event injection' may be a better way to do this in the end, but it seems to work pretty well already, and I think it shows how useful such a capability can be.
I'm all for it. Frederic, Steve, what do you think?
ITs really nice indeed, I'm reviewing the patches and will test them after weekend, it fits nicely with the idea of unifying top and report, i.e. the refresh thing he mentions could do the decay as well if a parameter is set.
Thnaks Frederic for pointing this patchset to me.
Great work!
- Arnaldo
Quoted 33 lines · Arnaldo Carvalho de Melo
Em Thu, Mar 04, 2010 at 12:18:56PM +0100, Ingo Molnar escreveu:
Quoted 24 lines · Tom Zanussi
Quoted 8 lines
Currently, a perf session entails two steps: first 'perf record' or 'perf trace record' records the perf data to disk, then 'perf report' or 'perf trace report' reads the saved data from disk and reports the results.
This experimental patchset makes some changes to perf that instead allow the perf data to be piped directly from the record step to the report step, without ever touching the disk.
Very nice!
Quoted 4 lines
Obviously, it would be better to have a real top-like display for these rather than a continuously scrolling mode like this, and of course it will be much more useful once we get the syscall name injection events going (the column on the left shows syscall numbers only).
It's still useful for ad-hoc tracing!
Side-note, it might make sense to expose the 'clear' escape sequence somehow in an easy fashion:
[H[2J
Quoted 7 lines
to make it non-scrolling ;-) via a pre-provided method.
Quoted 3 lines
There are some rough edges and inefficiencies, and I guess 'event injection' may be a better way to do this in the end, but it seems to work pretty well already, and I think it shows how useful such a capability can be.
I'm all for it. Frederic, Steve, what do you think?
Quoted 8 lines
ITs really nice indeed, I'm reviewing the patches and will test them after weekend, it fits nicely with the idea of unifying top and report, i.e. the refresh thing he mentions could do the decay as well if a parameter is set.
Thnaks Frederic for pointing this patchset to me.
Great work!
Thanks! Just FYI, I have a new version of this patchset that I'm hoping to post in the next couple of days. It adds the refresh sequence and a specifiable delay, among other things (like bugfixes)...
Tom
Quoted 2 lines
- Arnaldo
Quoted 176 lines · Tom Zanussi
Currently, a perf session entails two steps: first 'perf record' or 'perf trace record' records the perf data to disk, then 'perf report' or 'perf trace report' reads the saved data from disk and reports the results.
This experimental patchset makes some changes to perf that instead allow the perf data to be piped directly from the record step to the report step, without ever touching the disk.
This is especially useful for 'perf trace' - adding this capability means that the trace scripts are no longer relegated to simple post-processing, but can be run in a continuous 'live mode', forever processing the event stream and e.g. periodically dumping current results, essentially becoming special-purpose 'top' applications, or continuously scanning the event stream for arbitrarily complex conditions to flag, etc...
Being able to feed the event stream over a pipe also makes it possible to do things like trace over the network using e.g. netcat.
It turns out that perf is pretty close to being able to do this already, with the exception of the header data; most of the work of this patchset deals with changing that. It does so in a mainly additive way: it doesn't make any changes to the existing disk format or normal disk-mode processing, just adds special treatment for the cases when 'perf [trace] record' records to stdout or 'perf [trace] report reads from stdin.
Here are some quick examples. Basically using '-' as the filename to the -o and -i options send/read the data from stdout/stdin:
Here's a simple 'perf record' | 'perf report' run:
root@tropicana:~# perf record -o - -c 1 -f -a -M -R -e raw_syscalls:sys_enter | perf report -i - ^C# Samples: 120234 # # Overhead Command Shared Object Symbol # ........ ............... ......................... ...... # 98.65% perf libpthread-2.8.90.so [.] __read 0.46% perf libpthread-2.8.90.so [.] __write_nocancel 0.32% perf libpthread-2.8.90.so [.] __open_nocancel 0.29% perf libpthread-2.8.90.so [.] __libc_close 0.07% perf libc-2.8.90.so [.] 0x000000000a4b15 0.03% perf libc-2.8.90.so [.] __fxstat64 0.03% perf libc-2.8.90.so [.] __open 0.03% perf libc-2.8.90.so [.] close 0.02% Xorg libc-2.8.90.so [.] __select 0.01% Xorg libpthread-2.8.90.so [.] __read 0.01% perf libpthread-2.8.90.so [.] __libc_lseek 0.01% Xorg libc-2.8.90.so [.] setitimer 0.01% perf libc-2.8.90.so [.] __xstat64 0.01% firefox libpthread-2.8.90.so [.] __read 0.00% perf libc-2.8.90.so [.] llseek 0.00% Xorg libpthread-2.8.90.so [.] __restore_rt 0.00% Xorg libc-2.8.90.so [.] sigprocmask 0.00% Xorg libc-2.8.90.so [.] __poll 0.00% wterm libc-2.8.90.so [.] __read 0.00% gnome-screensav libpthread-2.8.90.so [.] __read 0.00% firefox libc-2.8.90.so [.] __poll 0.00% Xorg libc-2.8.90.so [.] writev 0.00% gnome-settings- libpthread-2.8.90.so [.] __read 0.00% wterm libc-2.8.90.so [.] __select 0.00% perf libc-2.8.90.so [.] statfs 0.00% mysqld libc-2.8.90.so [.] __select 0.00% firefox libpthread-2.8.90.so [.] __pthread_cond_timedwait 0.00% firefox libpthread-2.8.90.so [.] __lll_unlock_wake 0.00% firefox libpthread-2.8.90.so [.] 0x0000000000decb 0.00% firefox libc-2.8.90.so [.] __select 0.00% perf libc-2.8.90.so [.] getcwd 0.00% perf libc-2.8.90.so [.] mmap64 0.00% gnome-settings- libc-2.8.90.so [.] __poll 0.00% gnome-screensav libc-2.8.90.so [.] __poll 0.00% firefox libc-2.8.90.so [.] writev 0.00% apache2 libpthread-2.8.90.so [.] __waitpid 0.00% apache2 libc-2.8.90.so [.] __select # # (For a higher level overview, try: perf report --sort comm,dso) #Included in this patchset are a couple of 'top' scripts, rwtop and sctop, that are essentially just modifications of existing scripts. Basically the original scripts were modified to add a 5 second timer. In the handler for the timer, the current output summary is printed, and the state is cleared and begun anew, ad infinitum.
Here are the new scripts as shown in the perf trace list:
root@tropicana:~# perf trace -l List of available trace scripts: workqueue-stats workqueue stats (ins/exe/create/destroy) wakeup-latency system-wide min/max/avg wakeup latency rw-by-file <comm> r/w activity for a program, by file rwtop system-wide r/w top failed-syscalls [comm] system-wide failed syscalls rw-by-pid system-wide r/w activity syscall-counts-by-pid [comm] system-wide syscall counts, by pid failed-syscalls-by-pid [comm] system-wide failed syscalls, by pid sctop [comm] syscall top syscall-counts [comm] system-wide syscall countsAnd here's a few iterations of the output of the 'sctop' Python script:
root@tropicana:~# perf trace record sctop | perf trace report sctop perf trace started with Python script /root/libexec/perf-core/scripts/python/sctop.pysyscall events:
event count
---------------------------------------- -----------
2 179
3 1580 79 78 24 5 13 1 3 16 2 298 1 137 1 79 1 72 1 8 1syscall events:
event count
---------------------------------------- -----------
0 659287
202 19951 812 2 246 3 224 78 56 12 51 5 23 23 22 8 18 7 18 4 9 38 6 61 5 254 4 20 2 16 2 137 1 15 1 14 1 9 1syscall events:
event count
---------------------------------------- -----------
0 647178
202 1209
1 324
12 52
7 2914 6 16 4 23 3 3 3 72 2 21 1 11 1 9 1 5 1 4 1 2 1.
.
.
Quoted 6 lines
Obviously, it would be better to have a real top-like display for these rather than a continuously scrolling mode like this, and of course it will be much more useful once we get the syscall name injection events going (the column on the left shows syscall numbers only).
That's a very nice thing. I guess that we could make the scripting API to provide something to write such top like things.
Providing a simple dict must be enough, something organized like this:
topdict = {
"colname" : [val1, val2, val3],
"col2name" : [val4, val5, val6]
}Or may be a callback that returns a dict.