perf record: Allow poll timeout to be specified
PATCH 1/1 perf record: Allow poll timeout to be specified
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
--- tools/perf/Documentation/perf-record.txt | 6 ++++++ tools/perf/builtin-record.c | 5 ++++- tools/perf/perf.h | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)
tools/perf/Documentation/perf-record.txt +6 -0
@@ -250,6 +250,12 @@ is off by default. --running-time:: Record running and enabled time for read events (:S) +--poll=:: +Polling interval in msec. Defaults to infinite which means record relies on +watermarks to wakeup and read events from each mmap. Setting poll helps smooth +the event collection across mmaps and the subsequent processing of the data +file. For example perf-top uses a 100 msec polling interval. + SEE ALSO -------- linkperf:perf-stat[1], linkperf:perf-list[1]
tools/perf/builtin-record.c +4 -1
@@ -485,7 +485,7 @@ static int __cmd_record(struct record *rec, int argc, const char **argv)if (hits == rec->samples) {if (done || draining)break ; - err =perf_evlist__poll (rec->evlist,-1 ); + err =perf_evlist__poll (rec->evlist, opts->poll_timeout);/* * Propagate error, only if there's any. Ignore positive * number of returned events and interrupt error.
@@ -734,6 +734,7 @@ static struct record record = { .user_freq = UINT_MAX, .user_interval = ULLONG_MAX, .freq = 4000, + .poll_timeout = -1, .target = { .uses_mmap = true, .default_per_cpu = true,
@@ -841,6 +842,8 @@ struct option __record_options[] = { "Sample machine registers on interrupt"),OPT_BOOLEAN (0 ,"running-time" , &record.opts.running_time,"Record running/enabled time of read (:S) events" ), +OPT_INTEGER (0 ,"poll" , &record.opts.poll_timeout, +"poll interval in ms (defaults to infinite)" ), OPT_END() };
tools/perf/perf.h +1 -0
@@ -62,6 +62,7 @@ struct record_opts {u64 user_interval;bool sample_transaction;unsigned initial_delay; +int poll_timeout; };struct option ;
Signature · 1 line
2.2.1
Quoted 11 lines · David Ahern
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
Quoted 4 lines
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Please tune the default value (perhaps influenced by N_PROC?) so that users will get sane behavior without having to specify this option!
Thanks,
Ingo
Quoted 14 lines · Ingo Molnar
Quoted 11 lines · David Ahern
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
Quoted 7 lines
Quoted 4 lines
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Please tune the default value (perhaps influenced by
N_PROC?) so that users will get sane behavior without having to specify this option!
I knew you were going to say that! ;-)
It's really a function of events coming in not cpus. The number of CPUs just compounds the problem.
I thought about making perf-record use a 100msec timeout like perf-top, but that can lead to unnecessary FINISHED_ROUND events in the file and unnecessary noise/overhead in the record side. On the other hand looking at scheduler tracepoints, kvm tracepoints, etc -- those can flood in to the point that even 100msec is too long.
Arnaldo Carvalho de Melo
maintainerEm Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:
Quoted 12 lines · David Ahern
Quoted 11 lines
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
Quoted 7 lines
Quoted 4 lines
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Please tune the default value (perhaps influenced by
N_PROC?) so that users will get sane behavior without having to specify this option!
Isn't this a followup patch? I.e. changing the default from infinity to some sane value?
Applying it now.
- Arnaldo
Quoted 20 lines · Arnaldo Carvalho de Melo
Em Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:
Quoted 19 lines · David Ahern
Quoted 17 lines
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Quoted 5 lines
Quoted 3 lines
Please tune the default value (perhaps influenced by
N_PROC?) so that users will get sane behavior without having to specify this option!Isn't this a followup patch? [...]
Will a followup patch be written?
Thanks,
Ingo
Arnaldo Carvalho de Melo
maintainerEm Wed, Mar 25, 2015 at 10:11:47AM +0100, Ingo Molnar escreveu:
Quoted 21 lines · Arnaldo Carvalho de Melo
Quoted 19 lines
Em Tue, Mar 24, 2015 at 05:12:11PM +0100, Ingo Molnar escreveu:
Quoted 18 lines · David Ahern
Quoted 17 lines
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Please tune the default value (perhaps influenced by
N_PROC?) so that users will get sane behavior without having to specify this option!Isn't this a followup patch? [...]
Will a followup patch be written?
Hope so :-)
If David doesn't come up with something I'll probably will, as making 'trace' use the ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...
But what I said is independent of if a followup patch would come or not, right now we don't have that possibility, with his patch, we do.
Turning it from not possible to possible looks an improvement before we get it done automatically, and even by then allowing someone to tweak that value may be useful, no?
- Arnaldo
Quoted 5 lines · Arnaldo Carvalho de Melo
If David doesn't come up with something I'll probably will, as making 'trace' use the
ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...
I have thought about it. It needs to be an adaptive algorithm:
1. start at 100 msec.
2. Read the maps. How much data are there (not events, but data size)? 3. Adjust poll timeout up or down with some heuristic -- maybe something similar to the algorithm perf-top uses for removing entries from the histograms.
That said, I still thinking giving the user control is not a crazy idea.
David
Quoted 7 lines · David Ahern
Quoted 5 lines · Arnaldo Carvalho de Melo
If David doesn't come up with something I'll probably will, as making 'trace' use the
ordered_samples, like 'perf top' does (initially with some arbitrary reasonable poll timeout value), is a low hanging fruit to get those multi-CPU tracepoints sorted until I get something better in place...
Quoted 8 lines
I have thought about it. It needs to be an adaptive algorithm:
1. start at 100 msec.
2. Read the maps. How much data are there (not events, but data size)? 3. Adjust poll timeout up or down with some heuristic -- maybe something similar to the algorithm perf-top uses for removing entries from the histograms.That said, I still thinking giving the user control is not a crazy idea.
The user has absolutely zero control over the rate of data and about what the ideal refresh rate would be! So this really should be solved in software and automatically.
'try to tweak this parameter up and down until the tool is usable' is a really, really poor user interface concept ...
Thanks,
Ingo
Quoted 9 lines · David Ahern
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
so you have pipe to perf script, right?
Quoted 5 lines
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
there's also the '--no-buffering' option that sets:
attr->watermark = 0;
attr->wakeup_events = 1;but that's just the other edge, which is not what you'd want
I think it's good to have user side configurable as well
thanks,
jirka
Quoted 11 lines · Jiri Olsa
Quoted 9 lines · David Ahern
Record currently wakes up based on watermarks to read events from the mmaps and write them out to the file. The result is a file that can have large blocks of events per mmap before a finished round event is added to the stream. This in turn affects the quantity of events that have to be passed through the ordered events queue before results can be displayed to the user. For commands like perf-script this can lead to long unnecessarily long delays before a user gets output. Large systems (e.g, 1024 cpus) further compound this effect. I have seen instances where I have to wait 45 minutes for perf-script to process a 5GB file before any events are shown.
Quoted 1 line
so you have pipe to perf script, right?
$ perf record .... $ perf script ... <wait an eternity> data
Quoted 7 lines
Quoted 6 lines
This patch adds an option to perf-record to allow a user to specify the poll timeout in msec. For example using 100 msec timeouts similar to perf-top means the mmaps are traversed much more frequently leading to a smoother analysis side.
Quoted 6 lines
there's also the '--no-buffering' option that sets:
attr->watermark = 0; attr->wakeup_events = 1;but that's just the other edge, which is not what you'd want
right, that is the other extreme. record would never go to sleep.
Quoted 8 lines
I think it's good to have user side configurable as well
Acked-by Jiri Olsa <[email protected]>thanks,
jirka